How to Inspect and Trace Storage I/O Queue Operations in Linux Using blktrace and blkparse

When investigating database query slowdowns, storage throughput degradation, or erratic disk latency on Linux enterprise systems, high-level utilities such as iostat or vmstat provide only coarse averages. They cannot reveal whether delays occur while waiting in kernel block device queues, during bio-to-request merges, or inside the physical storage controller hardware. The Linux kernel contains an event tracing mechanism within the block layer that captures every low-level transaction. The official diagnostic toolset, blktrace and its companion parser blkparse, provides microsecond-level instrumentation of storage queues, revealing how I/O operations are scheduled, merged, and completed across physical drives.

Understanding the Lifecycle of a Block I/O Request

Whenever a process writes to or reads from disk, the kernel passes data through distinct stages in the block layer:

• Remap (A): An I/O is remapped to a different device or partition (e.g. inside LVM or software RAID).
• Queue (Q): An I/O request enters the generic block layer queue.
• Get Request (G): A request structure is allocated for the I/O.
• Merge (M): An incoming bio structure is merged with an existing adjacent request in the queue to optimize throughput.
• Dispatch (D): The completed request is dispatched from the I/O scheduler queue to the physical device driver.
• Complete (C): The hardware storage controller confirms execution completion back to the kernel.

Installing blktrace and Prerequisites

Install the blktrace suite using your distribution package manager:

On Debian and Ubuntu

sudo apt update
sudo apt install blktrace

On Fedora, RHEL, and AlmaLinux

sudo dnf install blktrace

Verifying debugfs Mounting

blktrace communicates with kernel event buffers via the debugfs filesystem. Verify that it is mounted:

mount | grep debugfs

If not present, mount it manually:

sudo mount -t debugfs debugfs /sys/kernel/debug

Recording Block Layer Traces with blktrace

Identify your target block storage device (e.g., /dev/nvme0n1 or /dev/sda) using lsblk. Because tracing interacts directly with physical hardware buffers, administrative privileges are required.

1. Capturing a Live Device Trace

To record storage activity on /dev/sda for 10 seconds, execute:

sudo blktrace -d /dev/sda -w 10

The utility captures binary event streams across all available CPU cores, outputting files formatted as sda.blktrace.<cpu_number>.

2. Piping Live Data Directly to blkparse

To inspect events in real time without storing multi-gigabyte binary trace files on disk, pipe blktrace directly into blkparse:

sudo blktrace -d /dev/sda -o - | blkparse -i -

Deciphering blkparse Output and Action Codes

Running blkparse against recorded data files generates a detailed event ledger:

blkparse -i sda

Sample output lines:

  8,0    2        1     0.000000000 14850  Q  WS 2048576 + 16 [kworker/u8:2]
  8,0    2        2     0.000000850 14850  G  WS 2048576 + 16 [kworker/u8:2]
  8,0    2        3     0.000002100 14850  I  WS 2048576 + 16 [kworker/u8:2]
  8,0    2        4     0.000010400 14850  D  WS 2048576 + 16 [kworker/u8:2]
  8,0    2        5     0.000340200     0  C  WS 2048576 + 16 [0]

Anatomy of a Trace Record:

• 8,0: Major and minor device numbers.
• 2: CPU core that handled the transaction.
• 1: Sequential event sequence index.
• 0.000000000: Timestamp relative to trace initialization.
• 14850: Process ID (PID) generating the I/O.
• Action Code (Q, G, I, D, C): Queued, Generated, Inserted into queue, Dispatched to driver, Completed by hardware.
• WS: Operation type (Write, Synchronous). Other flags include R (Read), A (Readahead), F (FUA/Flush).
• 2048576 + 16: Starting sector address followed by the block sector length.

Calculating Hardware Latency vs Queue Latency

The transition between action codes reveals where performance bottlenecks originate:

• Queue Delay (Q to D): The time elapsed between an I/O being queued and dispatched to hardware. Long Q-to-D latency indicates the storage scheduler is saturated or throttling requests under heavy write contention.
• Hardware Service Time (D to C): The time elapsed between dispatching to the storage driver and hardware completion. Long D-to-C latency indicates physical disk latency, degraded SSD NAND wear, or controller thermal throttling.

Generating Executive Summary Reports with btt

To avoid manually calculating timestamps across millions of lines, use the Block Trace Tool (btt) included with the blktrace suite to generate aggregated statistical profiles:

btt -i sda.blktrace.*

btt tabulates comprehensive metrics including average queue depths, seek distance patterns (evaluating sequential vs random access), and min/avg/max latencies for every phase of the block I/O pipeline.

Utilizing blktrace and blkparse delivers definitive, kernel-level storage diagnostics, enabling Linux engineers to pinpoint storage latency causes with forensic precision.