How to Trace Page Cache Readahead Latency and Efficiency in Linux Using eBPF

Disk I/O latency is a primary factor governing performance in storage-heavy Linux workloads, including analytical query engines, sequential file streaming services, and database index scans. To minimize physical disk seeks, the Linux kernel employs an intelligent heuristic mechanism known as page cache readahead. When an application reads data sequentially, the kernel anticipates future read requests and pre-fetches contiguous pages from storage into RAM before user space asks for them. Tracing readahead behavior and latency using eBPF enables systems engineers to verify whether readahead algorithms are accelerating workloads or causing memory thrashing.

The Mechanics of Linux Page Cache Readahead

When an application issues a read() or pread() system call, the kernel inspects the page cache. If the requested file offset is not in memory, a cache miss occurs, and physical storage must be queried. Concurrently, the kernel’s readahead subsystem (governed by page_cache_sync_readahead() and page_cache_async_readahead()) evaluates the process’s access patterns.

If sequential reading is detected, the kernel scales up its readahead window size (up to the filesystem limit, traditionally 128KB–256KB) and issues asynchronous background read requests to the underlying block device. However, if an application performs random access across large files (e.g., B-tree database lookups), aggressive readahead wastes I/O bandwidth and pollutes the page cache with pages that are evicted before being read.

Prerequisites and System Requirements

  • Linux host running kernel 5.4 or newer with eBPF and BPF Type Format (BTF) compiled in.
  • Root or CAP_BPF privileges.
  • The bpftrace and bpfcc-tools packages installed.
# Debian / Ubuntu installation
sudo apt update && sudo apt install bpftrace bpfcc-tools linux-headers-$(uname -r)

# RHEL / Rocky Linux installation
sudo dnf install bpftrace bcc-tools kernel-devel-$(uname -r)

Tracing Readahead Requests and Window Sizing in Real Time

To capture when the kernel initiates readahead operations and inspect the requested window size (in pages), run this bpftrace one-liner:

sudo bpftrace -e '
#include <linux/fs.h>

kprobe:page_cache_sync_ra {
    printf("%s [SYNC_RA]  Comm: %-16s | PID: %-6d | Requested: %lu pages
",
        time("%H:%M:%S"), comm, pid, arg2);
}

kprobe:page_cache_async_ra {
    printf("%s [ASYNC_RA] Comm: %-16s | PID: %-6d | Requested: %lu pages
",
        time("%H:%M:%S"), comm, pid, arg2);
}'

The output differentiates between:

  • SYNC_RA: Synchronous readahead invoked when the application was blocked waiting for an initial cache miss.
  • ASYNC_RA: Asynchronous readahead triggered while the application was already consuming previously pre-fetched pages, indicating seamless background streaming.

Measuring Readahead Execution Latency

To measure the physical block I/O time taken by the storage subsystem to satisfy readahead requests, trace the latency between readahead issuance and completion:

sudo bpftrace -e '
kprobe:ondemand_readahead {
    @start[tid] = nsecs;
}

kretprobe:ondemand_readahead {
    $start = @start[tid];
    if ($start) {
        @readahead_lat_us = hist((nsecs - $start) / 1000);
        delete(@start[tid]);
    }
}

END {
    clear(@start);
}'

Terminating this probe with Ctrl + C generates a logarithmic histogram showing readahead latency in microseconds, exposing whether underlying storage arrays are introducing latency bottlenecks.

Evaluating Readahead Efficiency and Unused Page Waste

The BCC tool collection provides a specialized high-level utility named readahead that measures the efficiency of readahead requests by tracking whether pre-fetched pages were actually accessed before eviction:

sudo readahead-bpfcc 10

After sampling for 10 seconds, the tool reports:

Total readahead pages: 18450
Readahead hit ratio:   94.2%
Unused pages evicted:  1070

A hit ratio above 85–90% indicates effective pre-fetching. Conversely, a hit ratio below 50% demonstrates that readahead is mispredicting access patterns and burdening disk queues unnecessarily.

Remediation and Tuning Recommendations

  • Adjust Block Device Readahead: Inspect and tune the maximum readahead sector limit on your block storage using blockdev:
    # Check current readahead sectors (typically 256 sectors = 128KB)
    sudo blockdev --getra /dev/nvme0n1
    
    # Increase for sequential streaming workloads (e.g., 1024 sectors = 512KB)
    sudo blockdev --setra 1024 /dev/nvme0n1
    
    # Decrease or set to 0 for random transactional databases
    sudo blockdev --setra 64 /dev/nvme0n1
  • Advise with madvise(): In software development, pass MADV_WILLNEED for known sequential ranges or MADV_RANDOM for random index access to inform the kernel readahead engine directly.

Get the best tech tips delivered straight to your inbox.

Join thousands of readers mastering Apple, Google, Microsoft, and Linux.