How to Use the sar and sysstat Commands in Linux for Historical System Activity Reporting

When a production Linux server experiences an unexpected slowdown, out-of-memory crash, or CPU lockup in the middle of the night, real-time diagnostic tools like top or htop cannot help after the fact. Diagnosing intermittent server anomalies requires historical performance telemetry. The sysstat suite and its primary reporting engine sar (System Activity Reporter) operate as background kernel performance collectors, capturing hardware metrics across CPU, RAM, disk I/O, swap, and network interfaces at regular 10-minute intervals and storing them in binary log archives.

Installing and Enabling the sysstat Service

On Debian and Ubuntu distributions, the sysstat package is installed with data collection disabled by default to save disk resources:

sudo apt update
sudo apt install sysstat -y

Enable data collection by modifying the sysstat configuration file:

sudo nano /etc/default/sysstat

Change ENABLED="false" to ENABLED="true". Then enable and start the systemd collector services:

sudo systemctl enable --now sysstat
sudo systemctl enable --now sysstat-collect.timer
sudo systemctl enable --now sysstat-summary.timer

sysstat Component	Operational Role	Storage / Target Location
sa1 (Collector)	Collects binary system metrics every 10 minutes.	Appends to daily binary log /var/log/sysstat/saDD.
sa2 (Reporter)	Generates daily text summaries before midnight.	Writes human-readable report /var/log/sysstat/sarDD.
sar (Query CLI)	Extracts and formats metric curves for specific intervals.	Interactive terminal output.

Querying Historical CPU Performance

By default, executing sar without arguments displays CPU metrics captured throughout the current day at 10-minute intervals:

sar -u

Output columns include %user, %system, %iowait, and %idle. If a server slowed down at 03:30 AM, examine the %iowait column; high I/O wait indicates disk saturation rather than CPU starvation.

To inspect CPU consumption for individual cores rather than system-wide averages:

sar -u -P ALL

Auditing Memory Utilization and Swap Thrashing

To check whether an application leak exhausted RAM and triggered the Linux OOM Killer:

# Inspect historical memory utilization
sar -r

# Inspect swap space activity and page-ins/page-outs
sar -S

Key metrics to evaluate:

• %memused: Total percentage of physical RAM consumed (excluding buffers and caches).
• %swpused: Percentage of swap partition allocated. Sudden jumps in swap usage paired with high pswpin/s indicate memory thrashing.

Analyzing Block Device and Disk I/O Saturation

Disk storage bottlenecks are a leading cause of database query timeouts. Query physical storage device metrics using the -d and -p flags:

sar -d -p

Examine the %util column for your primary drives (e.g., sda or nvme0n1). Disk utilization approaching 100% indicates that the storage controller is saturated with pending read/write queues.

Restricting Analysis to Specific Time Windows and Past Dates

You can pinpoint exact incident windows using the start (-s) and end (-e) time filters in 24-hour format:

# Query CPU metrics specifically between 03:00 and 04:00 AM today
sar -u -s 03:00:00 -e 04:00:00

# Inspect disk I/O from the 15th day of the current month
sar -d -p -f /var/log/sysstat/sa15

Configuring sysstat and mastering sar ensures Linux system administrators possess the historical forensic data required to perform definitive post-mortem incident analyses.