How to Use the iotop and iostat Commands in Linux to Diagnose Disk I/O Bottlenecks

When a Linux server grinds to a halt, displaying sluggish terminal responsiveness and soaring load averages, administrators frequently attribute the degradation to CPU starvation or memory exhaustion. However, running standard tools like top often reveals that CPU utilization is largely idle, yet the system is paralyzed by a high percentage of I/O wait (%wa). In modern computing, storage devices—whether mechanical hard drives, SATA SSDs, or high-speed NVMe flash arrays—possess finite queue depths and read/write bandwidth. When unoptimized database queries, uncontrolled log flushes, or rogue background processes saturate the storage subsystem, the Linux kernel places processes into un-interruptible sleep states (state D), locking system performance. To diagnose and neutralize these bottlenecks, systems administrators rely on two complementary utilities: iostat for disk-level performance auditing and iotop for process-level I/O tracing.

Comparing iostat and iotop Diagnostic Scopes

Diagnosing disk latency requires auditing both the physical storage hardware and the specific software processes generating I/O queues. The table below delineates the distinct operational roles of iostat and iotop:

Diagnostic Utility	Monitoring Scope	Primary Telemetry Provided	Typical Operational Role
iostat	Storage block devices (sda, nvme0n1) and partitions.	IOPS (r/s, w/s), throughput (MB/s), queue sizes (aqu-sz), and device utilization (%util).	Determines whether storage hardware platters or flash controllers are physically saturated.
iotop	Individual processes, threads, and user accounts.	Real-time read/write bandwidth, I/O priority classes, and percentage of time spent in disk sleep.	Pinpoints the exact Process ID (PID) or script responsible for generating excessive disk operations.

Installing Storage Diagnostic Tooling

Both utilities are available across standard Debian, Ubuntu, and Red Hat package repositories. Install them using:

sudo apt update && sudo apt install -y sysstat iotop

The sysstat package provides iostat along with background performance logging daemons.

Auditing Hardware Storage Telemetry with iostat

Executing iostat without arguments prints cumulative statistics calculated since the operating system was last booted. Because historical averages obscure immediate spikes, administrators must run iostat with sampling intervals.

1. Executing Extended Device Audits

To inspect real-time performance every second across active storage disks, use the extended statistics flag (-x) and omit inactive zero-activity devices (-z):

iostat -xz 1 5

This command records 5 sequential iterations sampled at 1-second intervals. The console displays a granular performance matrix:

Device            r/s     w/s     rMB/s     wMB/s   rrqm/s   wrqm/s  %rrqm  %wrqm  r_await  w_await   aqu-sz  rareq-sz  wareq-sz  svctm  %util
nvme0n1         14.00 1250.00      0.12     48.50     0.00   410.00   0.00  24.70     0.45     8.20     3.40      8.57     39.73   0.22  42.00
sda              0.00  320.00      0.00      4.10     0.00    12.00   0.00   3.61     0.00    65.40    14.20      0.00     13.12   3.12  99.80

2. Deciphering Key Performance Indicators (KPIs)

To identify where the storage bottleneck lies, evaluate these critical metrics:

Metric Column	Technical Meaning	Diagnostic Evaluation Threshold
r/s and w/s	Read and write requests issued per second (IOPS).	Shows current operational load compared to hardware IOPS limits.
rMB/s and wMB/s	Megabytes read or written per second (Throughput).	Indicates whether bus transfer bandwidth (SATA 6Gbps or PCIe lanes) is saturated.
r_await and w_await	Average time (in milliseconds) for I/O requests to be serviced.	Read/write latency. Values exceeding 20ms on mechanical HDDs or 5ms on NVMe SSDs indicate severe storage queuing.
aqu-sz	Average Request Queue Length.	Number of pending requests waiting in device queues. Values substantially above 1 indicate requests are backing up.
%util	Device bandwidth saturation percentage.	On mechanical disks, 100% indicates full platter saturation. On modern multi-queue NVMe drives, drives can sustain 100% util while processing parallel queues.

Tracing Offending Processes with iotop

Once iostat confirms that a storage device is saturated, use iotop to isolate the rogue binary or container.

1. Launching Interactive Process Monitoring

Because querying kernel task I/O structures requires root capabilities, launch iotop with elevated permissions:

sudo iotop

To filter out hundreds of idle background threads and display strictly the processes actively reading or writing to disk, press o (or launch with -o):

sudo iotop -o

The interactive display reveals:

• TID / PID: The specific thread identifier and process ID.
• PRIO: The assigned I/O priority class and level (e.g., be/4).
• USER: The system user account executing the binary (e.g., mysql, root, www-data).
• DISK READ / DISK WRITE: Real-time transfer rates in KB/s or MB/s.
• SWAPIN: Percentage of time the process is stalled waiting for memory pages to be retrieved from swap storage.
• IO>: The critical metric—percentage of time the process spent stalled in kernel I/O wait.

2. Tracking Accumulated I/O Consumption with -a

Burst-prone processes (such as log rotating scripts or backup dumps) fire intermittently, often escaping notice during instantaneous snapshots. To track cumulative bandwidth consumed over time, run:

sudo iotop -a -o

This accumulates total megabytes read and written since iotop was launched, making rogue batch jobs immediately apparent.

3. Headless Logging to File

To capture process activity in non-interactive server environments over a 10-minute diagnostic window:

sudo iotop -b -n 60 -d 10 -o -t > /tmp/iotop_diagnostic.log &

Remediating Disk Bottlenecks with ionice

Linux incorporates the ionice scheduling utility, allowing administrators to allocate I/O priority classes so that background operations (such as backups or indexing) never starve latency-critical databases or web services.

1. The Three I/O Scheduling Classes

• Class 1 (Real Time – rt): Grants highest priority access to storage queues before any other process. Use with extreme caution.
• Class 2 (Best Effort – be): Standard default scheduling class, supporting priority levels from 0 (highest) to 7 (lowest).
• Class 3 (Idle): The process receives zero disk access whenever any other application requires I/O.

2. Restricting Background Tasks to Idle Class

To force a heavy backup script or directory compression task to execute only when the storage controller has zero competing requests:

sudo ionice -c 3 tar -czf /backup/archive.tar.gz /var/www

To alter the priority of an existing running process (e.g., PID 4812):

sudo ionice -c 2 -n 7 -p 4812

This demotes the process to lowest-priority Best Effort, preventing it from degrading system responsiveness.