How to Trace Block Device I/O Latency in Linux Using biolatency

Storage subsystem bottlenecks represent one of the most frequent causes of application latency and system unresponsiveness in Linux production servers. Traditional disk monitoring utilities like iostat report average I/O service times, throughput, and queue utilisation over coarse sampling intervals. However, averages frequently obscure critical latency outliers: a database transaction that normally completes in 0.5 milliseconds might occasionally stall for 250 milliseconds due to storage queue contention or device lockup.

To capture and visualise the full distribution of block device I/O service times without suffering the performance penalties of older tracing utilities, Linux performance engineers rely on biolatency. Powered by extended Berkeley Packet Filters (eBPF), biolatency measures the exact duration between block I/O request issuance and hardware completion, rendering logarithmic latency histograms in real time. This tutorial demonstrates how to install, configure, and use biolatency to diagnose storage latency bottlenecks in Linux.

Understanding Block Device Latency and biolatency Architecture

When an application performs a filesystem read or write that cannot be fulfilled by the Linux page cache, the Virtual Filesystem (VFS) and block layer translate the request into one or more struct request objects dispatched to the underlying storage device driver.

Block I/O latency consists of two distinct components:

• Queue Latency: The time a request spends waiting in the device scheduler queue (e.g., BFQ, MQ-Deadline, or Kyber) before hardware dispatch.
• Device Service Time: The duration required by the physical NVMe, SSD, or rotational disk controller to read/write the requested sectors and return a hardware interrupt.

biolatency attaches eBPF kprobes and tracepoints to kernel block layer functions (including blk_account_io_start and blk_account_io_done). By calculating the timestamp delta between request initiation and hardware completion directly inside kernel space, biolatency constructs low-overhead latency distribution histograms with negligible CPU overhead.

Step 1: Install BCC Tools on Linux

biolatency is distributed as part of the BPF Compiler Collection (BCC) tools suite. Ensure your kernel version is 4.9 or later with eBPF support enabled.

On Ubuntu and Debian:

sudo apt update
sudo apt install -y bpfcc-tools linux-headers-$(uname -r)

On Debian and Ubuntu systems, BCC utilities append the -bpfcc suffix (e.g., biolatency-bpfcc). To run tools without the suffix, add the BCC tools path to your shell environment:

echo 'export PATH=$PATH:/usr/sbin' >> ~/.bashrc
source ~/.bashrc

On RHEL, Rocky Linux, and Fedora:

sudo dnf install -y bcc-tools kernel-devel-$(uname -r)
export PATH=$PATH:/usr/share/bcc/tools

On Arch Linux:

sudo pacman -S bcc bcc-tools linux-headers

Step 2: Trace System-Wide Block I/O Latency

To record disk I/O requests and print a summary latency histogram, run biolatency with root privileges:

sudo biolatency-bpfcc

Let the tool run for several seconds or minutes while workload activity occurs, then press Ctrl + C to terminate the trace. biolatency outputs a logarithmic histogram:

Tracing block device I/O... Hit Ctrl-C to end.
^C
     usecs               : count     distribution
         0 -> 1          : 0        |                                        |
         2 -> 3          : 0        |                                        |
         4 -> 7          : 12       |*                                       |
         8 -> 15         : 184      |*******                                 |
        16 -> 31         : 940      |****************************************|
        32 -> 63         : 421      |*****************                       |
        64 -> 127        : 58       |**                                      |
       128 -> 255        : 14       |                                        |
       256 -> 511        : 3        |                                        |
       512 -> 1023       : 0        |                                        |

In this example, the vast majority of block I/O requests completed between 16 and 31 microseconds (typical for high-performance NVMe solid-state storage), with a minor long-tail distribution reaching 511 microseconds.

Step 3: Measure Latency in Milliseconds

While microsecond precision (usecs) is ideal for fast NVMe storage, diagnosing latency spikes on rotational disks (HDDs), network-attached storage (NFS), or cloud virtual block volumes (e.g., AWS EBS) is easier when viewed in milliseconds. Pass the -m flag to switch units to milliseconds:

sudo biolatency-bpfcc -m

     msecs               : count     distribution
         0 -> 1          : 1420     |****************************************|
         2 -> 3          : 85       |**                                      |
         4 -> 7          : 12       |                                        |
         8 -> 15         : 4        |                                        |
        16 -> 31         : 1        |                                        |

Any requests registering in the >16 ms bins indicate significant I/O queuing or hardware throttling.

Step 4: Separate Disk Queuing Time from Device Service Time

When investigating storage latency, a crucial question arises: Is the physical drive slow, or are requests backing up inside the Linux kernel I/O scheduler queue?

By default, biolatency measures total time (queue latency plus device time). Pass the -Q flag to isolate and measure queuing time only:

# Measure only time spent waiting in OS queues
sudo biolatency-bpfcc -Q -m

• If biolatency -Q reveals high queuing latency (e.g., > 10 ms), the operating system is sending more concurrent I/O requests than the device queue depth can handle. Consider tuning your I/O scheduler or switching to none for fast NVMe drives.
• If queuing latency is negligible but standard biolatency shows high values, the physical drive hardware itself is bottlenecked.

Step 5: Break Down Latency by Storage Device and Operation Type

Production servers frequently attach multiple storage drives (e.g., an OS root disk and separate database data volumes). Group latency distributions across individual devices and operation types:

Display Latency Per Disk Device

Add the -d flag to output separate histograms for each block device (e.g., nvme0n1, sda, sdb):

sudo biolatency-bpfcc -d

Display Periodic Interval Summaries

To output a fresh histogram every 10 seconds for 6 iterations without manual cancellation:

sudo biolatency-bpfcc 10 6

This allows administrators to monitor latency trends continuously during database benchmarks, backup routines, or simulated traffic surges.