How to Trace Linux Page Cache Hit Ratios and Dirty Pages Using cachestat

The Linux kernel relies heavily on the system page cache to accelerate filesystem input/output operations by caching frequently accessed disk blocks and file metadata in unused physical RAM. While conventional utilities such as free, vmstat, and top reveal total memory allocated to buffers and cached pages, they provide no visibility into the actual efficiency of the cache—specifically whether running processes are hitting active memory pages or incurring costly block device disk reads. The cachestat tool, part of the BPF Compiler Collection (BCC), solves this monitoring limitation by attaching eBPF tracing probes directly to internal Linux kernel page cache functions. This article explains how to install BCC tools, trace page cache hit ratios in real time, monitor dirty page creation, and diagnose filesystem I/O performance bottlenecks using cachestat.

Understanding Linux Page Cache Architecture and eBPF Instrumentation

When an application requests file data through system calls such as read() or pread64(), the Linux Virtual File System (VFS) and page management subsystems check whether the target file pages are already resident in physical memory. If present, a page cache hit occurs, returning data at RAM bus speeds without triggering mechanical disk or NVMe storage transactions. If absent, a page cache miss occurs, requiring synchronous or asynchronous block I/O to fetch data into the cache. Similarly, write operations mark memory pages as “dirty” before background kernel flusher threads (such as kswapd and writeback daemons) synchronise dirty buffers back to persistent storage.

Traditional monitoring tools cannot compute page cache hit ratios because tracking every page lookup using standard profiling creates prohibitive overhead. cachestat leverages Linux extended Berkeley Packet Filters (eBPF) to trace specific low-overhead kernel probe points with negligible system impact, including:

• add_to_page_cache_lru(): Traced to measure page cache additions resulting from disk reads and misses.
• mark_page_accessed(): Traced to count page cache hits when an active resident page is referenced.
• account_page_dirtied(): Traced to capture pages modified by write operations awaiting background storage flushes.
• mark_buffer_dirty(): Traced across block layer operations to track dirty filesystem buffer headers.

Installing BCC Tools on Debian, Ubuntu, and Enterprise Linux

Before launching cachestat, verify that your Linux distribution provides BCC packages compiled against your running kernel headers. On Debian and Ubuntu systems, install the core BPF collection using apt:

sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

On Red Hat Enterprise Linux, AlmaLinux, Rocky Linux, and Fedora systems, install the corresponding BCC utilities using dnf:

sudo dnf install bcc-tools kernel-devel-$(uname -r)

On Ubuntu and Debian systems, BCC binaries are installed with a distribution suffix into /usr/sbin/cachestat-bpfcc, whereas Enterprise Linux environments place them under /usr/share/bcc/tools/cachestat. Create a convenient symbolic link or verify access via your system shell:

which cachestat-bpfcc || ls -l /usr/share/bcc/tools/cachestat

Tracing Live Page Cache Hit Ratios Using cachestat

To begin real-time page cache tracing across the entire operating system, execute cachestat with superuser privileges. By default, running the utility without arguments captures system activity and prints a performance row every second:

sudo cachestat-bpfcc

To specify custom sampling intervals and capture counts, pass the interval duration in seconds followed by the total number of iterations. For example, to sample page cache throughput every 5 seconds for 12 iterations (spanning one minute of observation), execute:

sudo cachestat-bpfcc 5 12

A representative terminal output displays the following columnar metrics:

    HITS   MISSES  DIRTIES    HIT%   BUFFERS_MB  CACHED_MB
   14820       12        4   99.9%          128       4192
   18402       48       22   99.7%          128       4193
    8912     6210      412   58.9%          128       4210
   24110       15       10   99.9%          129       4212

Understanding each reporting field ensures accurate diagnostic conclusions:

• HITS: The total number of page cache lookups satisfied entirely from existing memory pages during the sampling window.
• MISSES: The count of requested pages that were not cached in RAM, forcing the kernel to initiate block reads from underlying storage media.
• DIRTIES: The number of newly written pages modified in RAM that have not yet been flushed to disk.
• HIT%: The computed percentage of successful page cache hits relative to total lookups (HITS / (HITS + MISSES) * 100). High-performance transactional workloads typically sustain hit ratios above 95%.
• BUFFERS_MB: Physical memory allocated to raw disk block buffers, matching /proc/meminfo Buffer metrics.
• CACHED_MB: Total physical RAM dedicated to cached file pages.

Adding Timestamps and Correlating I/O Workloads

When diagnosing intermittent application slowdowns, correlating page cache misses with cron tasks, database backups, or web server spikes requires precise temporal logging. Use the -T flag to include a timestamp column in the terminal stream:

sudo cachestat-bpfcc -T 2

This command outputs timestamped telemetry suitable for redirection into operational audit logs:

TIME                  HITS   MISSES  DIRTIES    HIT%   BUFFERS_MB  CACHED_MB
14:10:02             21400       18        6   99.9%          256       8192
14:10:04              3120     8450     1200   26.9%          256       8220
14:10:06             29450       42       18   99.9%          257       8224

In the example above, a sudden drop in HIT% to 26.9% at 14:10:04 directly coincides with thousands of page misses, indicating that an unbuffered bulk read or cold cache query saturated the storage subsystem.

Diagnosing Low Hit Ratios and Dirty Page Bottlenecks

When cachestat reveals persistently depressed hit percentages (below 80%) or massive spikes in dirty page counts, investigate the following system-level root causes:

1. Insufficient Physical Memory: If total cached memory is constrained because resident processes consume all available RAM, the kernel aggressively evicts inactive cached pages. Check current memory pressure using vmstat 1 or inspect page faults with sar -B.

2. Kernel Dirty Ratio Tuning: When writing large sequential datasets, if dirty pages accumulate faster than storage controllers can flush them, write system calls block synchronously. Inspect your kernel writeback thresholds using sysctl:

sysctl vm.dirty_background_ratio vm.dirty_ratio

On high-throughput database servers, lowering these thresholds ensures smoother background I/O flushes without stalling interactive processes.

3. Uncached Batch Operations: Utilities such as rsync, database dumps (e.g. mysqldump), or backup scanners can rapidly evict useful working sets. Where appropriate, configure backup tools with posix_fadvise flags (such as POSIX_FADV_DONTNEED) to prevent cold backup reads from polluting the primary page cache.