How to Use the Linux sar Command to Monitor Historical System Performance

The Limitation of Real-Time Monitoring

When a Linux server experiences a sudden spike in CPU usage or runs out of RAM, administrators instinctively type top or htop into the terminal. These tools are excellent for viewing the exact state of the system at this very second.

However, server crashes rarely happen when you are actively watching the terminal. They usually happen at 3:00 AM on a Sunday. If a web server crashes overnight and automatically reboots, looking at top on Monday morning is completely useless. The system looks perfectly healthy now, and the critical data showing what caused the crash has evaporated from RAM.

To perform true forensic analysis, you need a tool that constantly records system metrics in the background and saves them to the hard drive for later review. That tool is the System Activity Reporter, known via its command-line interface as sar.

Step 1: Installing and Enabling sysstat

The sar command is part of the sysstat package. On many Linux distributions, it is installed but disabled by default.

1. Install it:
   • Ubuntu/Debian: sudo apt install sysstat
   • CentOS/RHEL: sudo yum install sysstat
2. Enable data collection:
   • Open the configuration file: sudo nano /etc/default/sysstat
   • Change ENABLED="false" to ENABLED="true".
   • Save and exit (Ctrl+O, Enter, Ctrl+X).
3. Start the service:
   • sudo systemctl enable sysstat
   • sudo systemctl start sysstat

By default, a background cron job will now wake up every 10 minutes, silently record the CPU, RAM, Disk I/O, and Network statistics, and save them into highly compressed binary files in /var/log/sysstat/.

Step 2: Using sar for Real-Time Analysis

Before looking at historical data, it helps to understand the basic sar syntax. You can use sar as a real-time monitor by specifying an interval (in seconds) and a count.

For example, to check the CPU usage every 2 seconds, for a total of 5 times, type:

sar 2 5

The output will display the percentage of CPU time spent in %user (applications), %system (kernel), and %idle. The final line will provide an Average.

Step 3: Traveling Back in Time

Now, let’s solve the Monday morning mystery. You know the server crashed on the 14th of the month at around 3:00 AM.

By default, if you just type sar, it displays the data from the beginning of today (midnight to now). To look at a previous day, use the -f (file) flag to point to the specific log file.

The files in /var/log/sysstat/ are named saDD, where DD is the day of the month.

sar -f /var/log/sysstat/sa14

This will print the entire CPU history of the 14th, in 10-minute increments. You can scroll up to the 3:00 AM timestamp to see if the CPU spiked to 100% (0% idle) right before the crash.

Step 4: Querying Specific Subsystems

A server doesn’t only crash due to CPU overload. Memory exhaustion (the Out of Memory killer) or disk bottlenecks are equally common.

You can tell sar to look at different hardware subsystems using specific flags, and combine them with the historical file flag.

• Memory (-r): sar -r -f /var/log/sysstat/sa14
  This will show kbmemfree (free RAM) and %memused. If you see memory usage slowly creeping up to 99% over several hours before the crash, you have identified a memory leak.
• Disk I/O (-b): sar -b -f /var/log/sysstat/sa14
  This displays TPS (transactions per second) and data read/written. A massive spike here indicates a heavy database query or a runaway backup script froze the system.
• Network (-n DEV): sar -n DEV -f /var/log/sysstat/sa14
  This shows network traffic by interface. A massive, sudden spike in incoming packets (rxpck/s) right before a crash is a strong indicator of a Denial of Service (DDoS) attack.

By mastering the sar command, you transition from reacting to live emergencies to becoming a forensic investigator, capable of determining exactly what killed a Linux server hours or days after the event occurred.