The Limitations of Syslog
When investigating a security breach on an Ubuntu server, administrators typically turn to the standard system logs (/var/log/syslog or /var/log/auth.log). However, these logs are entirely reactive and highly superficial. They can tell you that a user logged in via SSH, or that the Nginx service failed to start.
Standard logs cannot tell you who opened a specific classified text file, which process modified the /etc/passwd file, or how a piece of malware executed a specific system call to escalate privileges. If a hacker gains root access, they can simply delete the syslog files, erasing their tracks completely.
To achieve forensic-grade security, enterprise Linux engineers deploy auditd (The Linux Audit Daemon). auditd operates at the core of the Linux kernel. It intercepts system calls before they are fully processed. It can monitor every single read, write, or execution event on the server, bind it to a specific User ID (even if the user used sudo to switch to root), and write the evidence to a heavily protected log file that cannot be easily tampered with. If syslog is the security camera in the lobby, auditd is the microchip tracking every single keystroke in the vault.
Step 1: Installation and the Daemon Architecture
auditd is not installed by default on standard Ubuntu deployments. You must install the daemon and the associated user-space utilities.
sudo apt update
sudo apt install auditd audispd-plugins -yOnce installed, the daemon starts automatically. The configuration files are located in /etc/audit/. The two most critical files are:
auditd.conf: Controls the daemon itself (log file size, rotation, and action to take if the hard drive fills up).audit.rules: The actual rule engine that tells the kernel what to intercept.
Step 2: Defining File and Directory Watches
The most common use case for auditd is watching critical files for unauthorized modifications.
Suppose you want to know exactly who modifies the /etc/passwd (user database) or /etc/shadow (password hashes) files.
You can inject rules dynamically into the running kernel using the auditctl command. (To make them permanent, you add them to the audit.rules file).
sudo auditctl -w /etc/passwd -p wa -k identity_changes
sudo auditctl -w /etc/shadow -p wa -k identity_changesDecoding the Syntax:
-w /etc/passwd(Watch): Tells the kernel to monitor this specific file.-p wa(Permissions): Specifies the actions to trigger on.wstands for Write,astands for Attribute change (like changing permissions withchmod). You can also userfor Read andxfor Execute.-k identity_changes(Key): This is a custom tag. Because the audit log will generate millions of lines of data, assigning a key allows you to quickly filter the logs later.
Step 3: Monitoring System Calls (The Advanced Level)
Watching files is powerful, but advanced malware operates entirely in memory, making system calls directly to the kernel to escalate privileges or spawn reverse shells.
You can instruct auditd to monitor specific system calls across the entire operating system.
For example, you want to log every single time a user (other than the root user) attempts to use the execve system call (which is the kernel command used to execute a program). This will essentially create a forensic record of every single command typed by standard users.
sudo auditctl -a always,exit -F arch=b64 -S execve -F auid>=1000 -F auid!=4294967295 -k user_commandsDecoding the Syntax:
-a always,exit: Append the rule and trigger it when the system call exits (finishes).-S execve: The specific system call to monitor.-F auid>=1000: Only trigger if the Audit User ID (the original user who logged in) is 1000 or greater (meaning human users, ignoring system daemons).-F auid!=4294967295: Ignores events where the Audit ID is unset.
Step 4: Interrogating the Audit Log (ausearch)
The raw audit logs are stored in /var/log/audit/audit.log. However, the data format is highly cryptic, consisting of raw kernel hexadecimal values. You should never try to read this file using cat or grep.
Instead, use the dedicated ausearch utility, which parses the raw data and translates it into human-readable text.
Suppose you want to see all the events that triggered your “identity_changes” key from Step 2.
sudo ausearch -k identity_changes -iThe -i (Interpret) flag is critical. It converts raw hexadecimal strings into actual file paths and translates numeric UIDs (like 1001) into actual usernames (like jdoe).
To see all actions performed by a specific user within the last 60 minutes:
sudo ausearch -ua jdoe --start recent -iStep 5: Generating Management Reports (aureport)
If you are submitting a compliance report to an auditor, they do not want to see thousands of lines of terminal output. They want a summary.
The aureport command generates statistical summaries of the audit database.
To see a summary of all failed login attempts:
sudo aureport -l --failedTo see a statistical breakdown of every executable binary that has been run on the server, ranked by frequency:
sudo aureport -x --summaryThis command is incredibly powerful for identifying anomalies. If /bin/bash was executed 5,000 times, that is normal. If a strange binary named /tmp/payload.elf was executed once, it will stick out massively on the summary report, instantly alerting you to a potential compromise.
Conclusion
Relying on standard syslog for forensic investigation is a guarantee that sophisticated attackers will slip through the cracks. By integrating auditd deeply into the Ubuntu kernel, system administrators deploy an unbypassable, cryptographic surveillance engine. Whether you are tracking unauthorized file modifications to meet PCI-DSS compliance or monitoring execution system calls to hunt advanced malware, auditd provides the irrefutable evidence required to secure mission-critical Linux infrastructure.