Identifying System Bottlenecks
When a Linux server begins to stutter, lag, or drop database connections, the root cause is almost always resource starvation—specifically, the system has run out of physical RAM and is aggressively reading and writing to the swap file on the hard drive (a process known as “thrashing”).
While tools like htop and top provide excellent visual overviews of CPU usage, they are often too cluttered to quickly diagnose memory swapping issues. The vmstat (Virtual Memory Statistics) command provides a highly condensed, low-level snapshot of exactly how the Linux kernel is managing memory, swap, and CPU scheduling.
Running the vmstat Command
The vmstat utility is built into almost every Linux distribution by default. If you simply run the command by itself, it will output a single line of data representing the average statistics since the server was last booted.
vmstat
However, historical averages are useless when diagnosing an active performance crisis. To monitor the server in real-time, you should instruct vmstat to output a new line of data at a specific interval. For example, to output a new line every 2 seconds, run:
vmstat 2
The command will run continuously, printing a new row of data every two seconds until you stop it by pressing Ctrl + C.
Deciphering the Output
The output is divided into six distinct categories: Procs, Memory, Swap, IO, System, and CPU. When hunting for memory bottlenecks, you should focus entirely on the Memory and Swap columns.
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 524288 12480 45600 854000 1200 4500 4800 2100 1200 1500 45 15 30 10 0
The Memory Columns
- swpd: The amount of virtual memory (swap space on the hard drive) currently in use. If this number is consistently high, your server does not have enough physical RAM for its workload.
- free: The amount of completely idle RAM. (Note: In Linux, a low “free” number is actually normal and good, because the kernel aggressively uses spare RAM for caching).
The Swap Columns (The Danger Zone)
- si (Swap In): The amount of memory being read from the hard drive back into RAM (measured in KB/s).
- so (Swap Out): The amount of memory being moved from RAM out to the hard drive to free up space.
The Golden Rule: If you see the si and so columns consistently reading 0, your memory health is perfect. The system is operating entirely in physical RAM. However, if the si and so columns are showing hundreds or thousands of kilobytes per second (as seen in the example above), the server is actively thrashing. The hard drive cannot keep up with the read/write requests, causing the CPU wa (Wait for IO) metric to spike, and effectively bringing the server to a halt.