The Blind Spot of Real-Time Monitoring
When an Ubuntu server crashes at 3:00 AM and reboots automatically, the system administrator logs in at 9:00 AM to investigate. They run htop, iostat, and free -m. The server looks perfectly healthy. The CPU is at 5%, and RAM is abundant. However, this is real-time data. The administrator has absolutely no idea what the CPU utilization was at 2:59 AM, one minute before the crash.
Without historical telemetry, troubleshooting a post-mortem server crash is pure guesswork. You cannot fix an application if you cannot prove whether it ran out of memory, saturated the disk I/O, or flooded the network card.
To eliminate this massive blind spot, Linux engineers deploy the sysstat package, specifically the sar (System Activity Reporter) daemon. Unlike real-time tools, sar runs silently in the background via a cron job. Every 10 minutes, it takes a highly detailed, mathematical snapshot of every subsystem in the Linux kernel (CPU, RAM, Disk, Network, Swap) and writes it to a heavily compressed binary log file. If a server crashes, the administrator can use sar to effectively “rewind time” and view the exact state of the hardware in the minutes leading up to the disaster.
Step 1: Installing and Activating Data Collection
While the sysstat package is available in the Ubuntu repositories, installing it does not automatically enable the background data collection.
sudo apt update
sudo apt install sysstat -yAfter installation, you must explicitly instruct the daemon to begin recording data. Open the primary configuration file:
sudo nano /etc/default/sysstatChange the ENABLED flag from "false" to "true":
ENABLED="true"Save the file and restart the service:
sudo systemctl restart sysstat
sudo systemctl enable sysstatFrom this moment on, the sysstat cron job (located at /etc/cron.d/sysstat) will wake up every 10 minutes, capture the kernel metrics, and write them to daily binary files located in /var/log/sysstat/ (e.g., sa25 for the 25th day of the month).
Step 2: Analyzing Historical CPU Utilization
Suppose you are investigating a crash that occurred yesterday (the 24th of the month). You want to see the CPU utilization.
You use the sar command and point it to yesterday’s binary file using the -f (file) flag:
sar -f /var/log/sysstat/sa24The output is a massive table displaying the CPU usage in 10-minute intervals from midnight to midnight. Look at the %user (application CPU), %system (kernel CPU), and %iowait (CPU waiting on slow hard drives) columns.
If you only care about the time surrounding the crash (2:30 AM to 3:30 AM), you can aggressively filter the output using the -s (start) and -e (end) flags:
sar -s 02:30:00 -e 03:30:00 -f /var/log/sysstat/sa24If the %user column jumps from 15% to 99% at exactly 2:50 AM, you have definitively proven that a runaway application caused the crash.
Step 3: Analyzing Historical Memory and Swap Usage
Memory leaks are the most common cause of spontaneous server reboots (due to the Linux OOM Killer). To analyze RAM usage yesterday, you use the -r (RAM) flag:
sar -r -f /var/log/sysstat/sa24This will show you exactly how much free RAM (kbmemfree) was available, and how much RAM was used for caching (kbbuffers and kbcached).
Crucially, you must also check Swap utilization. If the server runs out of RAM, it attempts to use the hard drive as slow virtual memory (Swap). To view Swap statistics, use the -S flag:
sar -S -f /var/log/sysstat/sa24If you see %swpused climb from 0% at 1:00 AM to 100% at 2:58 AM, you have irrefutable proof that the server suffered a catastrophic memory leak.
Step 4: Analyzing Historical Network Traffic
If an application developer claims the server was disconnected from the network, you can prove whether the network card was actually passing traffic.
Use the -n DEV (Network Device) flag:
sar -n DEV -f /var/log/sysstat/sa24This provides an incredibly detailed breakdown of every single network interface (eth0, lo, docker0). It shows the exact number of packets received (rxpck/s), packets transmitted (txpck/s), and the exact bandwidth in kilobytes (rxkB/s, txkB/s). If the eth0 interface was pushing 50 Megabytes per second at the exact moment the developer claimed it was disconnected, the logs prove the developer is wrong.
Step 5: Adjusting the Polling Interval
By default, sysstat captures data every 10 minutes. In a highly volatile environment where a script might spike the CPU for only 2 minutes and then crash, the 10-minute polling window might completely miss the event.
To increase the resolution, you must edit the cron job.
sudo nano /etc/cron.d/sysstatChange the cron expression from 5-55/10 * * * * (every 10 minutes) to */2 * * * * (every 2 minutes).
(Note: Increasing the polling resolution will increase the size of the binary log files and consume slightly more CPU/I/O, but modern servers can easily handle a 2-minute interval).
Conclusion
Attempting to perform root-cause analysis on a crashed Linux server without historical telemetry is an exercise in futility. By enabling the sysstat data collection daemon and mastering the sar command, Ubuntu administrators can rewind time, interrogating precise, minute-by-minute historical records of CPU, Memory, and Network utilization to definitively prove why a server failed.