How to Analyze Linux Disk I/O Performance using the iostat Command

The Bottleneck of Disk Operations

When a Linux database server (like PostgreSQL or MySQL) begins to run slowly, system administrators often immediately blame the CPU or the RAM. They upgrade the server to 32 cores and 128GB of RAM, only to find the database is still crawling.

In reality, the most common performance bottleneck in modern servers is Disk I/O (Input/Output). If a database is trying to write 50,000 transactions per second to a physical hard drive, but the SSD array can only handle 10,000 writes per second, the entire application will stall, waiting for the disks to catch up. This state is known as “I/O Wait.”

To accurately diagnose whether your hard drives are the root cause of your server’s performance issues, you must analyze the disk subsystems using the iostat command.

Installing sysstat

The iostat utility is not always installed by default on minimal Linux distributions. It is bundled within the sysstat package.

To install it on Ubuntu/Debian:

sudo apt-get install sysstat

To install it on CentOS/RHEL:

sudo yum install sysstat

Running a Real-Time Analysis

Simply typing iostat and pressing Enter is not very useful, as it only gives you the average statistics since the server was last booted. To diagnose an active performance issue, you need a continuous, real-time feed of the disk metrics.

Execute the following command to receive an extended disk report every 2 seconds:

iostat -x -d 2

• -x: Displays extended statistics (crucial for finding bottlenecks).
• -d: Displays only device (disk) utilization, hiding the CPU summary to keep the screen clean.
• 2: The interval (in seconds) between each screen refresh.

Interpreting the Output (The Golden Metrics)

The iostat -x output is a massive wall of numbers. You only need to focus on three specific columns to diagnose an I/O bottleneck.

1. %util (Utilization Percentage)

This is the most critical metric on the screen. It is located in the far-right column. %util indicates how much time the disk spent actively reading or writing data versus sitting idle.

• If %util is at 5% to 30%, your disks are healthy and handling the load easily.
• If %util is hovering at 99% or 100%, your disk is entirely maxed out. It is the definitive bottleneck of your server, and no amount of CPU upgrades will fix the issue. You must migrate to faster NVMe storage.

2. await (Average Wait Time)

Located near the middle of the screen, await measures the average time (in milliseconds) that an I/O request had to sit in the queue waiting to be processed by the disk.

• An await time of 1ms to 5ms is excellent (typical for NVMe SSDs).
• If the await time spikes to 100ms or 500ms, it means the disk is so overwhelmed that database transactions are waiting half a second just to be written, causing massive application-level latency.

3. rkB/s and wkB/s (Throughput)

These columns show the raw data throughput—kilobytes read per second and kilobytes written per second. If you see a massive spike in wkB/s simultaneously with a 100% %util, you know a process is actively dumping a massive amount of data to the drive. You can then use tools like iotop to figure out exactly which process is causing the spike.

Press Ctrl+C to stop the continuous feed when you are finished analyzing.