Diagnosing intermittent storage performance issues on Linux servers is notoriously difficult. Traditional monitoring utilities like iostat or sar aggregate data over specific time intervals, providing a high-level overview of disk throughput and average wait times. While useful for general capacity planning, these tools are practically blind to microsecond-level latency spikes, often hiding the exact moment a specific process causes a storage bottleneck. To achieve true real-time visibility into kernel-level storage operations, Ubuntu administrators must turn to Extended Berkeley Packet Filter (eBPF) technology and the bcc-tools suite.
The Power of eBPF and BCC Tools
eBPF is a revolutionary technology built directly into the modern Linux kernel. It allows administrators to run sandboxed programs within the kernel space without requiring kernel modules or system reboots. This provides unprecedented visibility into operating system events with virtually zero performance overhead, making it perfectly safe for production environments.
The BPF Compiler Collection (BCC) is a toolkit that simplifies the development of eBPF programs and provides a comprehensive suite of pre-built tracing utilities. These tools allow you to instrument specific kernel functions and capture exact timing data for individual I/O requests.
Installing bcc-tools on Ubuntu
The bcc-tools package is available directly from the standard Ubuntu repositories. To install the toolkit and its dependencies, execute the following command:
sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)Once installed, the tracing scripts are located in the /sbin/ directory, typically suffixed with -bpfcc. Because eBPF interacts directly with kernel memory structures, all BCC tools must be executed with root privileges.
Tracing Latency Histograms with biolatency
If you suspect a disk is suffering from latency spikes, the most effective starting point is the biolatency tool. Rather than showing averages, it generates a real-time histogram of block device I/O latency, allowing you to instantly see if a small percentage of requests are taking unusually long to complete.
sudo biolatency-bpfccWhen you terminate the command (using Ctrl+C), it outputs an ASCII histogram showing the distribution of latency in microseconds. If the vast majority of operations complete within 500 microseconds, but a distinct secondary cluster of operations takes 50,000 microseconds (50 milliseconds), you have identified a severe, intermittent latency issue that traditional monitoring would have averaged out and hidden.
Identifying Culprit Processes with biosnoop
Once you confirm latency exists, you must identify the specific process causing or suffering from the delay. The biosnoop utility prints a real-time log of every single block device I/O operation, including the exact process name, process ID (PID), sector number, size, and the latency of that specific operation.
sudo biosnoop-bpfccThe output streams continuously. You can pipe this output into grep or awk to filter for specific processes or for latency values exceeding a certain threshold. For example, to only view operations taking longer than 10 milliseconds, you would filter the latency column.
Tracing Slower File System Operations with ext4slower
Block device tracing is powerful, but it lacks context regarding the actual files being accessed. If your Ubuntu server utilises the standard Ext4 file system, the ext4slower tool bridges this gap. It instruments the Virtual File System (VFS) layer and only outputs file operations (read, write, open, sync) that exceed a specific latency threshold, providing the exact filename being accessed.
sudo ext4slower-bpfcc 5This specific command filters out normal operations and only displays Ext4 file system calls that take longer than 5 milliseconds to complete. The output includes the process name and the exact file path. By correlating this data, you can definitively prove whether a database index rebuild, an aggressive backup script, or a specific application log file is the root cause of the storage bottleneck on your Ubuntu server.