How to Use the ‘sar’ Command for Historical System Performance Auditing

The Forensic Limitation of Top and Htop

When a Linux server experiences a sudden, catastrophic performance spike, the standard reaction is to run top or htop. While these tools are excellent for seeing what is happening right now, they are completely useless for seeing what happened five minutes ago.

If a rogue process spikes the CPU to 100% and then immediately terminates, you will never see it in top. To perform true forensic analysis and visualize historical system performance over time, Enterprise Linux administrators rely on the sar (System Activity Reporter) command.

1. Enabling the sysstat Daemon

Unlike top, sar is not a live monitor; it is a historical reporting engine. For it to work, you must install and enable the underlying data collector daemon (sysstat).

sudo apt install sysstat
sudo systemctl enable --now sysstat

Once enabled, the system quietly wakes up every 10 minutes in the background, takes a comprehensive snapshot of every hardware metric (CPU, RAM, Disk I/O, Network), and saves it to a binary log file in /var/log/sysstat/.

2. Generating a Historical CPU Report

Suppose you receive an alert that the database server crashed at 2:00 AM, but you didn’t log in until 8:00 AM. You can ask sar to replay the exact CPU usage from that time window.

sar -u -s 01:50:00 -e 02:10:00

Breaking down the flags:

• -u: Instructs sar to specifically report on CPU utilization.
• -s 01:50:00 (Start): Sets the exact start time for the report.
• -e 02:10:00 (End): Sets the exact end time for the report.

The output will be a highly structured table showing the exact CPU load, broken down by user-space, kernel-space (system), and I/O wait, in 10-minute intervals leading up to and during the crash.

3. Auditing Disk I/O Bottlenecks

CPU is rarely the actual bottleneck on modern servers; the culprit is usually storage I/O. If your web application is suddenly incredibly slow, you can use sar to audit the historical read/write speeds of your physical disks.

sar -d -p 1

Breaking down the flags:

• -d: Instructs sar to report on block device (disk) activity.
• -p: “Pretty-prints” the device names (e.g., displaying sda1 instead of obscure device mapper UUIDs).
• 1: Tells sar to generate a live, rolling report every 1 second, bypassing the historical logs.

Look at the %util (Utilization) column. If a disk consistently shows 100% utilization, your storage array is completely saturated, and adding more RAM or CPU to the server will not fix the performance issue.

4. Analyzing Network Saturation

If you suspect the server is under a Denial of Service (DoS) attack, you can use sar to audit historical network interface saturation.

sar -n DEV -f /var/log/sysstat/sa15

• -n DEV: Instructs sar to report on Network Device statistics.
• -f /var/log/sysstat/sa15: Forces sar to read from a specific historical file (e.g., the 15th day of the month), rather than the current day’s log.

You can see exactly how many kilobytes per second were transmitted (txkB/s) and received (rxkB/s) on eth0 during the exact minute the attack began.

Conclusion

The sar command is the definitive “flight data recorder” for Linux servers. By constantly polling hardware telemetry in the background and providing a powerful querying interface, it allows administrators to travel back in time and perform surgical post-mortem analyses on catastrophic performance failures.