How to Trace Block I/O Requests and Latency per Event in Linux Using biosnoop

Diagnosing erratic disk performance in Linux requires tracing storage requests from the instant a user-space application issues an I/O call down to the physical block device. While summary statistics like those from iostat illustrate average queue depths and throughput, they cannot attribute individual sluggish I/O operations to specific processes. The biosnoop utility, part of the BCC (BPF Compiler Collection) suite, bridges this gap by intercepting block layer kernel tracepoints to print a real-time event log for every disk read and write operation, complete with per-event latency, process name, and sector details.

Understanding How biosnoop Traces Storage Events

The Linux block I/O layer sits between high-level filesystems and physical device drivers. When an application writes data to disk, the kernel queues bio and request structures. Using eBPF (Extended Berkeley Packet Filter), biosnoop attaches probes to block device request submission (block_rq_issue) and completion (block_rq_complete) tracepoints.

When a request is submitted to the controller queue, biosnoop stores a high-precision timestamp. When the hardware driver signals completion, the utility calculates the elapsed duration, joins it with the invoking process ID (PID) and command name, and prints the completed transaction. Unlike biolatency, which summarises distributions into histogram buckets, biosnoop records an explicit stream of individual operations, making it straightforward to identify rogue background tasks stalling storage queues.

Installing Prerequisites and BCC Packages

To run biosnoop, your system requires matching kernel headers and the BCC eBPF toolchain. On Debian and Ubuntu systems, install the dependencies via apt:

sudo apt update && sudo apt install -y bpfcc-tools linux-headers-$(uname -r)

On Red Hat Enterprise Linux, Fedora, or Rocky Linux, use dnf:

sudo dnf install -y bcc-tools kernel-devel-$(uname -r)

On Debian/Ubuntu systems, the binary is accessible as biosnoop-bpfcc (or symlinked to biosnoop in /usr/sbin/). Root or sudo privileges are required to attach eBPF programs to kernel tracepoints.

Running biosnoop to Trace Real-Time Disk Activity

To begin monitoring block I/O events system-wide, run the utility without additional flags:

sudo biosnoop

The console prints a structured table that updates continuously as disk requests complete:

TIME(s)     COMM           PID    DISK    T SECTOR     BYTES  LAT(ms)
0.000000    jbd2/sda2-8    412    sda     W 4194304    4096      0.82
0.003412    mysqld         1842   nvme0n1 W 10485760   16384     1.45
0.015290    rsync          8920   sdb     R 20971520   131072    3.21
0.024102    kworker/u16:1  319    sda     W 4194320    8192      0.65

Each column conveys vital operational telemetry:

• TIME(s): Time elapsed in seconds since tracing commenced.
• COMM: Executable command name generating or submitting the I/O request.
• PID: Process ID associated with the transaction.
• DISK: Target block device node (e.g. sda, nvme0n1).
• T: Type of operation—R for Read, W for Write, or D for Discard/TRIM.
• SECTOR: Physical sector address on the storage media.
• BYTES: Total payload size in bytes.
• LAT(ms): Elapsed round-trip latency in milliseconds.

Adding Wall-Clock Timestamps for Log Correlation

When correlating disk stalls with enterprise monitoring alerts or application log files, relative seconds from script launch are difficult to cross-reference. Add the -t or -T flag to output wall-clock timestamps:

sudo biosnoop -T

With -T, the first column outputs TIME as HH:MM:SS (for instance, 14:22:08), allowing you to match a 200 ms disk delay directly to corresponding database query timeout entries.

Filtering by Storage Device and Queue Delays

On servers hosting multiple storage volumes—such as separate boot SSDs, database NVMe drives, and bulk HDD arrays—tracing every disk simultaneously can clutter terminal output. You can isolate trace collection to a specific storage controller or filter by latency thresholds.

If you suspect that excessive payload sizes are choking bus queues, pairing biosnoop event streams with bitesize request size histograms clarifies whether an application is issuing efficient large contiguous transfers or overwhelming the drive controller with millions of uncoalesced tiny writes.

Furthermore, if you need to trace execution paths within the filesystem itself rather than hardware queues, combining this analysis with funclatency or counting VFS layer calls with vfscount helps determine whether delays originate in software page cache routines or physical NAND flash controller stalls.