How to Trace Block Device I/O Pattern and Request Latency Using biosnoop in Linux

Storage subsystem bottlenecks represent one of the most common causes of high system load and unpredictable application latency in Linux servers. While aggregate tools like iostat or vmstat report general device utilisation percentages and average throughput, they cannot pinpoint which individual process initiated a disk read or write, nor can they measure latency per I/O transaction. The biosnoop utility, built upon eBPF within the BCC toolkit, traces every single block device request in real time.

How biosnoop Instruments the Linux Storage Subsystem

The biosnoop tool hooks into the Linux kernel block layer tracepoints, primarily block:block_rq_issue and block:block_rq_complete. By tracking the exact timestamp when a storage request is dispatched to the disk controller driver and when the hardware hardware interrupt returns completion, biosnoop calculates the round-trip latency of every individual block transaction.

Installing the BCC Tracing Utilities

Ensure that BCC tools and matching kernel headers are installed on your Linux distribution:

# On Debian, Ubuntu, or Pop!_OS:
sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

# On RHEL, AlmaLinux, Rocky Linux, or Fedora:
sudo dnf install bcc-tools kernel-devel-$(uname -r)

On Debian/Ubuntu environments, execute the tool as biosnoop-bpfcc. On enterprise RHEL environments, it is found at /usr/share/bcc/tools/biosnoop.

Running biosnoop to Capture Real-Time I/O Telemetry

To begin monitoring disk I/O requests across all storage block devices, run biosnoop with root privileges:

sudo biosnoop-bpfcc

Example terminal output:

TIME(s)     COMM           PID    DISK    T SECTOR     BYTES  LAT(ms)
0.000000    mysqld         1842   nvme0n1 W 2048590    16384     0.28
0.001420    jbd2/nvme0n1-8 421    nvme0n1 W 4194304    4096      0.15
0.045100    postgres       2190   sda     R 8388608    8192      8.42
0.052010    rsync          3011   sda     R 10485760   65536    14.80

Analysing Key Output Columns

Understanding each metric allows systems engineers to isolate root causes:

• COMM & PID: The process name and identifier that initiated the I/O request.
• DISK: The target block device (e.g., nvme0n1, sda, vda).
• T (Type): Indicates whether the operation was a Read (R) or Write (W).
• SECTOR: The starting storage sector number, useful for identifying sequential vs. random access patterns.
• BYTES: Payload size of the I/O request. Small, frequent transfers indicate fragmented I/O, whereas large blocks signify bulk sequential streaming.
• LAT(ms): The end-to-end device latency in milliseconds. High values on mechanical disks or SATA SSDs highlight queue saturation or queue starvation.

Diagnosing Common Storage Issues

Using biosnoop data, you can quickly identify and mitigate storage anomalies:

• Journaling Bottlenecks: If jbd2 (ext4 journal) accounts for continuous write latency spikes, consider tuning commit intervals (commit=60) or relocating the journal to a dedicated NVMe drive.
• Read Starvation: High read latencies (LAT > 10ms) occurring while large writes execute indicate inadequate I/O scheduling; switching from none to bfq or kyber can restore responsiveness.

By capturing granular block device transactions, biosnoop equips administrators with definitive evidence to diagnose and eliminate storage latency across Linux production infrastructure.