How to Trace Block I/O Queue Latency in Linux Using biolatency

Diagnosing erratic database query latency, slow filesystem writes, and application stalls in Linux frequently points to underlying storage subsystems. While standard tools like iostat report average I/O latency (await) over periodic intervals, averages conceal catastrophic latency outliers—a system averaging 1 millisecond per I/O can still experience occasional multi-second disk stalls that freeze transactional threads. The biolatency utility, part of the BCC (BPF Compiler Collection) suite, uses extended Berkeley Packet Filters (eBPF) to record the precise duration of every block I/O request, outputting logarithmic latency histograms.

Understanding Linux Block Layer Architecture and Latency

When an application reads or writes to a file, the request traverses the Virtual File System (VFS), the filesystem driver (e.g., ext4, XFS), and enters the Linux block layer. In modern multi-queue block layer architectures (blk-mq):

  • Requests are scheduled through hardware dispatch queues associated with individual CPU cores.
  • The request passes to the device driver (e.g., nvme, ahci, or scsi) and is transmitted to physical drive controllers.
  • Once the storage device completes the read or write, an interrupt signals the kernel, completing the request.

Total block device latency consists of two distinct components: time spent waiting in software dispatch queues, and actual physical device execution time. The biolatency tool instruments blk_account_io_start() and blk_account_io_done() to measure this end-to-end journey.

Prerequisites and Tool Installation

Running biolatency requires superuser privileges and an eBPF-capable kernel (version 4.9 or higher):

# Debian / Ubuntu installation
sudo apt update && sudo apt install bpfcc-tools linux-headers-$(uname -r)

# RHEL / Rocky Linux / AlmaLinux
sudo dnf install bcc-tools kernel-devel-$(uname -r)

On Debian/Ubuntu systems, the binary is available as /usr/sbin/biolatency-bpfcc.

Measuring Disk Latency Histograms in Real Time

To capture disk latency across all storage devices on your host, execute the utility without arguments:

sudo biolatency-bpfcc

Allow the tool to sample for 10 to 30 seconds during active disk workloads, then press Ctrl + C. The tool outputs a power-of-two histogram:

Tracing block device I/O... Hit Ctrl-C to end.
^C
     usecs               : count     distribution
         0 -> 1          : 0        |                                        |
         2 -> 3          : 0        |                                        |
         4 -> 7          : 12       |                                        |
         8 -> 15         : 840      |**                                      |
        16 -> 31         : 12450    |****************************************|
        32 -> 63         : 3210     |**********                              |
        64 -> 127        : 142      |                                        |
       128 -> 255        : 18       |                                        |
       256 -> 511        : 4        |                                        |
       512 -> 1023       : 1        |                                        |

In this NVMe example, the dominant mode is between 16 and 31 microseconds, demonstrating healthy storage performance without outliers.

Advanced Runtime Flags for Root Cause Analysis

To isolate which disks, operational flags, or processes are responsible for slow I/O, biolatency provides specialized flags:

1. Isolating Queuing Time from Device Time

By default, biolatency measures device execution time. To include software dispatch queue latency (time spent queued in the block elevator before transmission to hardware), pass the -Q flag:

sudo biolatency-bpfcc -Q

If latencies shift significantly higher with -Q, the operating system is generating I/O faster than the storage controller queue depth can absorb, pointing to software scheduler contention.

2. Separating Reads from Writes

Disk controllers frequently prioritize sequential reads over synchronous write flushes. To evaluate read and write performance independently, supply the -F flag:

sudo biolatency-bpfcc -F

The utility outputs two separate histograms: one for read operations and one for write operations.

3. Per-Disk Breakdown

On servers hosting multiple storage arrays or NVMe namespaces, isolate performance on a per-device basis using the -d flag:

sudo biolatency-bpfcc -d

Diagnostic Interpretations and Remediation

  • Bimodal Latency Distributions: A histogram showing two distinct peaks (e.g., one at 30 microseconds and another at 50 milliseconds) often indicates SSD internal garbage collection pauses or controller write-cache exhaustion.
  • Elevated Queue Latency (-Q): Check the active I/O scheduler in /sys/block/<device>/queue/scheduler. On modern NVMe drives, ensure the scheduler is set to none rather than legacy mq-deadline or bfq to prevent unnecessary software queuing overhead.
  • High Outlier Counts: Correlate latency spikes with filesystem sync operations using syncsnoop to determine if periodic fsync() calls are bottlenecking disk queues.

Get the best tech tips delivered straight to your inbox.

Join thousands of readers mastering Apple, Google, Microsoft, and Linux.