How to Trace VFS Directory Lookup Latency in Linux Using eBPF

Filesystem performance issues in complex Linux workloads—such as container runtimes with layered overlayfs mounts, source code compilation trees, and large web assets directories—often stem from directory path resolution bottlenecks rather than physical data read/write throughput. Every file access requires the Linux Virtual File System (VFS) to resolve pathname components (e.g., /var/log/nginx/access.log) step by step. When the directory entry cache (dcache) experiences high lock contention or misses, file opens stall. Tracing VFS directory lookup latency using eBPF provides deep observability into path traversal overhead.

Understanding VFS Path Walk and the Dentry Cache

When a process calls openat() or stat(), the kernel enters the VFS path lookup subsystem, primarily through filename_lookup() and link_path_walk(). For each path component, the kernel checks whether a corresponding directory entry (dentry) exists in memory in the dcache:

• RCU-Walk (Fast Path): The kernel traverses the dcache concurrently using Read-Copy-Update (RCU) locks without acquiring heavy spinlocks. This is extremely fast, executing in nanoseconds.
• Ref-Walk (Slow Path): If an entry is missing, invalid, or requires symbolic link traversal, the kernel falls back to ref-walk mode, acquiring reference counts and invoking the filesystem driver’s lookup() method to read directory blocks from disk.

In environments containing hundreds of thousands of files in a single folder, or deeply nested mount namespaces, directory lookup stalls can cascade across worker threads. Tracing vfs_lookup() reveals these latency outliers.

Prerequisites and Tool Installation

Tracing VFS kernel routines requires superuser privileges and an eBPF-enabled Linux kernel (5.4 or higher recommended):

# Ubuntu / Debian
sudo apt update && sudo apt install bpftrace bpfcc-tools linux-headers-$(uname -r)

# RHEL / Rocky Linux
sudo dnf install bpftrace bcc-tools kernel-devel-$(uname -r)

Tracing Directory Lookup Latency with bpftrace

To measure the execution time of vfs_lookup() calls and correlate high latency with specific directory entry names, execute this targeted bpftrace script:

sudo bpftrace -e '
#include <linux/fs.h>
#include <linux/dcache.h>

kprobe:vfs_lookup {
    @start[tid] = nsecs;
    @name[tid] = ((struct dentry *)arg1)->d_name.name;
}

kretprobe:vfs_lookup {
    $start = @start[tid];
    if ($start) {
        $lat_us = (nsecs - $start) / 1000;
        if ($lat_us > 1000) { // Log lookups taking over 1 millisecond
            printf("%s [SLOW LOOKUP] Comm: %-16s | Dentry: %-20s | Latency: %d us
",
                time("%H:%M:%S"), comm, str(@name[tid]), $lat_us);
        }
        delete(@start[tid]);
        delete(@name[tid]);
    }
}

END {
    clear(@start);
    clear(@name);
}'

The script captures lookups exceeding 1,000 microseconds (1 millisecond). Output lines expose the process name, target directory or file component name, and the latency duration.

Evaluating Dcache Hit Ratios Using dbsnoop / dcache-bpfcc

To assess overall directory cache health across the entire system, measure dcache hit rates using the dcstat tool from the BCC collection:

sudo dcstat-bpfcc 1

The utility outputs per-second dcache performance metrics:

TIME     REFS/s   SLOW/s   MISS/s   HIT%
06:01:10 45210    120      45       99.64%
06:01:11 48900    310      98       99.17%
06:01:12 84120    8940     4200     84.42%

A sustained drop in HIT% below 90% indicates severe dcache thrashing, often caused by automated backup scanners, sprawling container overlays, or insufficient memory allocated to slab caches.

Diagnostic Interpretations and Kernel Tuning

• VFS Lookups Slower than 10ms: Points to disk I/O waits during cold dcache lookups. Check underlying disk queue latency with iostat -xz 1.
• VFS Slab Reclaim Pressure: When memory pressure is high, the kernel reclaims dentries aggressively. You can inspect active dentry counts via cat /proc/sys/fs/dentry-state.
• Tuning vfs_cache_pressure: If your server has surplus RAM and should retain cached directory paths longer, adjust vm.vfs_cache_pressure in /etc/sysctl.d/99-vfs.conf:

  # Default is 100; lowering to 50 instructs the kernel to prefer retaining dentries over page cache
  sudo sysctl -w vm.vfs_cache_pressure=50