The Limitations of standard syslog
In a standard Ubuntu deployment, most administrative actions are recorded in /var/log/syslog or /var/log/auth.log. While sufficient for basic troubleshooting, these logs are entirely dependent on the application developer. If a malicious script running as www-data silently reads the /etc/shadow password file, or if a disgruntled administrator uses the rm command to delete proprietary source code, the standard syslog daemon will record absolutely nothing, because rm and cat do not proactively write to the system logs.
To achieve compliance with strict security frameworks (like HIPAA, PCI-DSS, or SOC 2), organizations must prove exactly who accessed what file, at what specific microsecond, and exactly which system call they executed to do it.
To achieve this forensic-level visibility, Ubuntu administrators deploy the auditd (Audit Daemon) package. auditd hooks directly into the Linux kernel. It does not rely on applications to report their activity. Instead, it intercepts and logs the raw system calls (syscalls) executed by the CPU, providing an irrefutable, cryptographically secure audit trail of all system activity.
Step 1: Installing and Enabling auditd
The auditd package is not installed by default on standard Ubuntu Server images.
sudo apt update
sudo apt install auditd audispd-plugins -yOnce installed, enable and start the service:
sudo systemctl enable auditd
sudo systemctl start auditdBy default, auditd immediately begins logging administrative events (like user logins and sudo executions) to its highly structured, unalterable log file located at /var/log/audit/audit.log.
Step 2: Configuring File and Directory Watches
The true power of auditd is its ability to “watch” specific, highly sensitive files. If anyone (even the root user) reads, writes, or executes a watched file, the kernel generates an immediate audit event.
To create a temporary rule that watches the /etc/passwd file for any Read (r), Write (w), Execute (x), or Attribute Change (a) events, use the auditctl command:
sudo auditctl -w /etc/passwd -p wa -k password-file-tamperingDecoding the Flags:
-w(Watch): The absolute path to the file or directory.-p wa(Permissions): Tells the kernel to only trigger the log if the file is Written to or its Attributes (permissions) change. We omit ‘r’ (read) because every single login attempt reads this file, which would flood the logs.-k(Key): A custom tag (“password-file-tampering”) that makes it infinitely easier to search the massive log file later.
If you want to watch an entire directory (like a folder containing proprietary SSH keys):
sudo auditctl -w /home/admin/.ssh/ -p rwxa -k ssh-key-accessStep 3: Creating Permanent Rules
Rules injected via auditctl are ephemeral; they are wiped from the kernel memory the moment the server reboots.
To make rules permanent, you must define them in the auditd configuration directory. On modern Ubuntu versions, do not edit /etc/audit/audit.rules directly. Instead, create a custom rules file inside the /etc/audit/rules.d/ directory.
sudo nano /etc/audit/rules.d/50-custom-watches.rulesAdd your watch rules to this file exactly as you typed them on the command line (minus the auditctl prefix):
-w /etc/shadow -p wa -k shadow-tampering
-w /var/www/html/secure_config.php -p rwx -k web-config-accessSave the file and instruct the daemon to compile and load the new ruleset:
sudo augenrules --loadStep 4: Interrogating the Audit Log (ausearch)
The raw /var/log/audit/audit.log file is exceptionally difficult for a human to read. It contains raw syscall numbers, hexadecimal strings, and encoded user IDs.
You must use the ausearch (Audit Search) utility to translate and filter the data.
To search the log for any events that triggered our custom password-file-tampering key, and explicitly translate the raw UIDs into human-readable usernames (-i for interpret):
sudo ausearch -k password-file-tampering -iIf an attacker successfully gained root access and modified the file, the output will definitively show:
- time: The exact microsecond the file was opened.
- syscall: The specific kernel command used (e.g.,
openat). - auid: The Audit User ID. This is critical. Even if the attacker used
sudoto become therootuser (UID 0), theauidtracks their original login ID (e.g., thedev-johnaccount), completely neutralizing the anonymity of the root account. - exe: The exact binary used to modify the file (e.g.,
/usr/bin/nanoor/bin/sed).
Step 5: Generating Analytical Reports (aureport)
If your security team demands a weekly summary of system activity, manually parsing logs is tedious. The aureport utility instantly crunches the massive audit log into concise, categorized tables.
To generate a summary report of all failed login attempts across the server:
sudo aureport --auth --failed -iTo see a strict summary of exactly which binary executables have been run on the server recently (invaluable for detecting unauthorized scripts):
sudo aureport --executable --summary -iConclusion
For organizations subjected to intense regulatory scrutiny, application-level logging is insufficient. By hooking directly into the Ubuntu kernel with auditd, deploying targeted file watches, and leveraging the ausearch utility to unmask privileged users, system administrators can build a forensic environment where no file access, configuration change, or unauthorized execution can occur without an unalterable, cryptographically secure record being generated.