How to Trace VFS File Read Latency Histograms in Linux Using eBPF

Diagnosing application performance degradation in Linux often leads systems engineers to examine block storage devices. However, looking exclusively at disk queue latency (await) ignores the entire operating system caching layer. Applications interact with storage through the Linux Virtual File System (VFS) via read(), pread64(), and readv() system calls. When a read hits the in-memory page cache, it returns in sub-microsecond time; when it misses, it blocks on physical storage or network filesystems. Tracing VFS read latency distributions using eBPF exposes both cache-hit performance and storage-bound read stalls in real time.

The Anatomy of a VFS Read Operation

When an application invokes the read() system call, execution transitions through several kernel architectural tiers:

1. VFS Entry (vfs_read()): The kernel verifies file descriptor permissions, validates memory offsets, and invokes the underlying filesystem’s specific file operations table.
2. Filesystem Dispatch: File systems like ext4 (ext4_file_read_iter()) or XFS query the page cache. If the requested data pages are resident, data is copied directly to user space memory.
3. Block Layer Fallback: If pages are not resident in RAM (cache miss), the filesystem constructs buffer heads or bios, submitting I/O requests down to the block layer and putting the calling thread to sleep until storage hardware responds.

Tracing at the vfs_read() layer captures total time experienced by user-space applications, providing an accurate metric of perceived storage responsiveness.

Prerequisites and Tool Installation

Tracing VFS functions requires root privileges and a kernel supporting extended Berkeley Packet Filters (eBPF) with BPF Type Format (BTF) compiled in (Linux 5.4 or higher recommended):

# Ubuntu / Debian
sudo apt update && sudo apt install bpftrace bpfcc-tools linux-headers-$(uname -r)

# RHEL / Rocky Linux / Fedora
sudo dnf install bpftrace bcc-tools kernel-devel-$(uname -r)

Generating VFS Read Latency Histograms with bpftrace

To measure the execution time of vfs_read() calls across all running processes and output a logarithmic latency histogram, deploy this high-performance bpftrace script:

sudo bpftrace -e '
#include <linux/fs.h>

kprobe:vfs_read {
    @start[tid] = nsecs;
}

kretprobe:vfs_read {
    $start = @start[tid];
    if ($start) {
        $lat_us = (nsecs - $start) / 1000;
        @vfs_read_latency_us = hist($lat_us);
        delete(@start[tid]);
    }
}

END {
    clear(@start);
}'

Allow the script to sample workload activity for 15 to 30 seconds, then press Ctrl + C. bpftrace prints a power-of-two histogram:

@vfs_read_latency_us: 
[0]                    4 |                                                    |
[1]                 1840 |@@@@@@@@@@@@@@@@@@@@@@@@                            |
[2, 4)              3920 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[4, 8)               812 |@@@@@@@@@@@                                         |
[8, 16)              140 |@                                                   |
[16, 32)              42 |                                                    |
[32, 64)              12 |                                                    |
...
[1024, 2048)          18 |                                                    |
[2048, 4096)           4 |                                                    |

This histogram reveals a bimodal distribution: the massive peak at 1–4 microseconds represents fast page cache hits, while the tail extending past 1024 microseconds (1 millisecond) represents physical storage read operations.

Isolating Outlier Latencies Per Process Name

To identify which specific applications are experiencing severe read latency stalls exceeding 10 milliseconds (10,000 microseconds):

sudo bpftrace -e '
kprobe:vfs_read {
    @start[tid] = nsecs;
}

kretprobe:vfs_read {
    $start = @start[tid];
    if ($start) {
        $lat_us = (nsecs - $start) / 1000;
        if ($lat_us > 10000) {
            printf("%s [SLOW READ] Comm: %-16s | PID: %-6d | Latency: %d ms
",
                time("%H:%M:%S"), comm, pid, $lat_us / 1000);
        }
        delete(@start[tid]);
    }
}

END {
    clear(@start);
}'

Diagnostic Interpretations and Remediation Strategies

• Dominant Tail Above 5ms: Confirms high page cache miss rates accompanied by slow physical storage response times. Inspect disk health with smartctl and verify underlying queue depths with iostat -xz 1.
• Elevated Latency on Small Files: In distributed filesystems (such as NFS, Ceph, or GlusterFS), high VFS read latency often stems from network round-trips for metadata attribute revalidation. Tune filesystem mount options (e.g., increasing actimeo on NFS mounts).
• Improving Cache Hit Ratios: Expand system page cache capacity by scaling physical RAM, or adjust vm.vfs_cache_pressure to prevent the kernel from evicting filesystem cache structures prematurely.