How to Use iostat in Linux to Diagnose Storage Latency, Queue Depths, and Disk I/O Bottlenecks

Architectural Foundation of the Linux Block Layer and iostat Telemetry

Storage subsystem bottlenecks represent one of the most critical degradation vectors in Linux servers, databases, and container clusters. When applications experience unresponsive latency or elevated uninterruptible sleep (D state) thread counts, storage I/O starvation is frequently the underlying culprit. The iostat utility, maintained within the open-source sysstat monitoring suite, provides granular observability into block device throughput, service latency, and kernel request queue saturation.

To accurately interpret iostat telemetry, systems engineers must understand how the Linux kernel processes storage operations. Whenever a process issues a read() or write() system call, the Virtual Filesystem (VFS) transmits block I/O requests (bio structures) down through filesystem caches into the Generic Block Layer. Here, multi-queue I/O schedulers (such as mq-deadline, kyber, or bfq) merge contiguous sectors, throttle aggressive threads, and dispatch requests into device submission queues. The kernel continuously tallies these lifecycle timings inside /proc/diskstats, which iostat parses and computes across sampled time intervals.

Metric Column	Kernel Counter Source	Measurement Unit	Technical Definition	Diagnostic Evaluation Baseline
r/s & w/s	Reads / Writes merged	Operations / sec	Completed read and write I/O requests per second (IOPS)	Workload baseline comparison
rMB/s & wMB/s	Sector count / 2048	Megabytes / sec	Data transfer throughput dispatched to the block device	SATA SSD (~500 MB/s); NVMe Gen4 (5000+ MB/s)
await	Wait time + Service time	Milliseconds (ms)	Average total time requests spent waiting in queue plus transfer time	NVMe: < 1 ms; Enterprise SSD: < 5 ms; HDD: < 20 ms
r_await & w_await	Read/Write separate wait	Milliseconds (ms)	Independent latency calculations for read versus write requests	Identifies whether read stalls or write flushes dominate
aqu-sz	Queue length integration	Requests (count)	Average number of I/O requests queued at the block device driver	Target < 1.0 on single drives; elevated values signify saturation
%util	Device busy ticks	Percentage (%)	Fraction of CPU time during which requests were issued to the device	Misleading on NVMe/RAID (multi-queue arrays saturate at >100% capacity)

Installing and Invoking iostat on Modern Linux Distributions

The iostat binary is packaged within the sysstat toolchain. On production servers, install and enable the persistent background data collection daemon:

# Debian / Ubuntu Server
sudo apt update && sudo apt install -y sysstat

# RHEL / Rocky Linux / AlmaLinux / Fedora
sudo dnf install -y sysstat

# Enable and start the sysstat background telemetry collector
sudo systemctl enable --now sysstat

Executing Granular Real-Time Diagnostics

When investigating real-time latency spikes, avoid running iostat without interval arguments. Invoking iostat once prints cumulative statistics since the last operating system boot, which masks immediate performance degradation.

Step 1: Inspect Extended Device Metrics Excluding Idle Disks

The -x flag exposes extended latency metrics, -z suppresses inactive drives with zero I/O activity, and -k or -m sets data units to Kilobytes or Megabytes:

# Monitor extended storage metrics every 1 second, printing timestamps
iostat -xzmt 1

Example telemetry output breakdown:

Device            r/s     w/s     rMB/s     wMB/s   rrqm/s   wrqm/s  %rrqm  %wrqm   r_await   w_await    aqu-sz    rareq-sz  wareq-sz  %util
nvme0n1         450.0   120.0     18.50      8.20      0.0     15.0    0.0   11.1      0.45      0.82      0.35       42.0      70.0  12.4%
sda              85.0   420.0      2.10     45.60      2.0    180.0    2.3   30.0     45.20    180.50      8.60       25.3     111.2  98.2%

In this output, nvme0n1 operates with healthy sub-millisecond response times (r_await 0.45 ms). Conversely, rotational drive sda demonstrates severe write saturation: w_await has escalated to 180.50 ms with an average queue length (aqu-sz) of 8.60 and a %util of 98.2%, indicating that write operations are backlogged in the queue.

Step 2: Isolating Disk Latency from CPU Utilisation

To evaluate whether I/O wait is causing CPU stall cycles, combine device telemetry with CPU utilisation:

# Monitor both CPU wait (%iowait) and block device latency every 2 seconds
iostat -c -x -t 2 10

If %iowait in the CPU block rises above 20% while await is elevated on your storage volume, your system is storage-bound. The CPU cores are actively waiting for disk controllers to complete I/O before thread execution can resume.

Advanced Diagnostics: Correlating Processes with Storage Pressure

Once iostat identifies the specific saturated physical disk or partition, trace the responsible processes using iotop and pidstat:

# 1. Trace individual process I/O read and write throughput in real-time
sudo iotop -oPa

# 2. Query per-process disk statistics with pidstat (sysstat suite)
pidstat -d 1 5

# 3. Check disk queue scheduler policy for the impacted device
cat /sys/block/sda/queue/scheduler
# [mq-deadline] kyber bfq none

# 4. Switch NVMe drives to 'none' scheduler to bypass software queuing overhead
echo none | sudo tee /sys/block/nvme0n1/queue/scheduler

Troubleshooting Common Storage Bottlenecks Identified by iostat

Observed iostat Anomaly	Underlying Root Cause	Technical Verification Method	Remediation Protocol
High await with low rMB/s & wMB/s	Extreme random I/O thrashing; small request sizes saturating drive seek times	Examine rareq-sz / wareq-sz (< 4 KB); check random seek latency	Align database tablespaces; implement write buffering or migrate to NVMe flash
Elevated w_await while r_await is normal	Filesystem journal flushing (ext4/xfs sync commits) or RAID battery-backed write cache disabled	Inspect sync calls via bcc-tools/biosnoop; query RAID controller cache mode	Enable Write-Back caching on hardware RAID controller; tune dirty_writeback_centisecs
%util at 100% on high-speed NVMe SSD	Misleading metric; %util measures time busy, ignoring parallel hardware queues	Examine await: if await remains < 1 ms despite 100% util, device is healthy	Rely on aqu-sz and await rather than %util for enterprise NVMe arrays
%util at 100% on high-speed NVMe SSD	Misleading metric; %util measures time busy, ignoring parallel hardware queues	Examine await: if await remains < 1 ms despite 100% util, device is healthy	Rely on aqu-sz and await rather than %util for enterprise NVMe arrays
Sudden surge in aqu-sz and latency across all disks	Memory exhaustion triggering aggressive kernel page scanning and dirty page throttling	Check vmstat 1; inspect si (swap-in) and so (swap-out) columns	Increase RAM allocations; lower vm.swappiness and adjust vm.dirty_ratio
Periodic 30-second await spikes	Database transaction log checkpointing or flushing of massive dirty page pools	Inspect /proc/sys/vm/dirty_background_bytes and DB checkpoint configurations	Smooth write commits by lowering dirty_background_ratio to flush data progressively