How to Trace Directory Cache Lookups and Misses in Linux Using dcsnoop

In high-performance Linux environments, filesystem path resolution forms the backbone of file input/output. Every time an application opens a file, traverses a directory hierarchy, or loads shared libraries, the kernel queries the Virtual Filesystem (VFS) Directory Cache (dcache) to translate path strings into inode numbers. When the dcache experiences heavy misses, the kernel must execute synchronous disk reads to load directory structures, causing severe I/O latency bottlenecks.

Diagnosing dcache thrashing and tracing filesystem lookups in real time is virtually impossible with conventional tools like iostat or vmstat. To inspect dcache lookup efficiency and identify the exact processes querying directory paths, Linux performance engineers rely on dcsnoop, an extended Berkeley Packet Filter (eBPF) tracing tool. This tutorial covers how to install, configure, and use dcsnoop to trace directory cache queries and diagnose path resolution latency in Linux.

Understanding the Linux Directory Cache (dcache) and dcsnoop

The Linux dcache maintains a hierarchical in-memory tree of dentry structures representing directory components and files. When an application requests /var/log/nginx/access.log, the kernel performs iterative lookups for each component (var, log, nginx, and access.log) within the dcache.

If a component is found in RAM, it is a cache hit (reference count updated with zero disk overhead). If missing, it results in a cache miss, triggering filesystem-specific lookup routines that access physical storage media.

dcsnoop attaches eBPF kprobes to kernel lookup functions (such as lookup_fast() and d_lookup()). It captures every lookup attempt, recording the requesting process, thread ID, query path, cache hit/miss status, and elapsed resolution latency without altering system runtime behaviour.

Step 1: Install BCC Tools on Linux

dcsnoop is packaged within the BPF Compiler Collection (BCC) tools suite. Ensure your system runs Linux kernel 4.9 or later with eBPF support enabled.

On Ubuntu and Debian:

sudo apt update
sudo apt install -y bpfcc-tools linux-headers-$(uname -r)

On Debian and Ubuntu, BCC utilities append the -bpfcc suffix (e.g., dcsnoop-bpfcc). To run tools seamlessly, add /usr/sbin to your path:

echo 'export PATH=$PATH:/usr/sbin' >> ~/.bashrc
source ~/.bashrc

On RHEL, Rocky Linux, and Fedora:

sudo dnf install -y bcc-tools kernel-devel-$(uname -r)
export PATH=$PATH:/usr/share/bcc/tools

On Arch Linux:

sudo pacman -S bcc bcc-tools linux-headers

Step 2: Basic Real-Time Directory Tracing with dcsnoop

To trace every dcache lookup across all active processes on the system, run dcsnoop with root privileges:

sudo dcsnoop-bpfcc

The tool immediately outputs active directory queries:

TIME(s)            PID    COMM             T [FAILED] PATH
0.000000000        2410   git              M          .git/config
0.000312040        2410   git              R          /etc/gitconfig
0.015240110        1820   nginx            R          /usr/share/nginx/html/index.html
0.021400200        3401   python3          M [FAIL]   lib/python3.10/site-packages/custom

Key output columns include:

• TIME(s): Elapsed seconds since tracing commenced.
• PID: The process ID performing the lookup.
• COMM: The process command name.
• T (Type): Indicates the lookup type:
  • R (Reference): Direct dcache reference hit (found in cache).
  • M (Miss): Cache miss (required disk inode lookup).
• [FAILED]: Displayed if the requested file or directory does not exist on disk.
• PATH: The directory or file component evaluated by the VFS layer.

Step 3: Filter by Cache Misses to Identify I/O Bottlenecks

When tuning web servers, databases, or build pipelines, cache hits complete in sub-microsecond intervals. Cache misses, however, force storage roundtrips. Pass the -m flag to filter the output, displaying only cache misses:

sudo dcsnoop-bpfcc -m

This isolates operations that bypassed memory caching, highlighting directories that cause disk thrashing under heavy multi-tenant workloads.

Step 4: Output Human-Readable Timestamps and Latency

To correlate directory cache spikes with server incidents, add timestamps and measure lookup latency:

Print Wall-Clock Timestamps

sudo dcsnoop-bpfcc -T

TIME     PID    COMM             T [FAILED] PATH
14:22:01 2410   find             M          /var/log/journal

Trace Lookup Latency in Milliseconds

Pass the -d flag to calculate how long each dcache lookup took to resolve:

sudo dcsnoop-bpfcc -d

Sample output:

TIME(s)            PID    COMM             T [FAILED] LAT(ms) PATH
0.000000000        5102   mysqld           M          1.84    /var/lib/mysql/data/ibdata1
0.004120100        1200   dockerd          R          0.02    /var/run/docker.sock

Lookups exhibiting several milliseconds of latency reflect storage queue contention or slow spinning disk seek times.

Step 5: Target a Specific Application Daemon

On active multi-service servers, system-wide directory tracing produces overwhelming log volumes. Filter lookups by process ID using the -p flag:

# Trace directory lookups exclusively for process ID 2840
sudo dcsnoop-bpfcc -p 2840

This allows performance engineers to profile build tasks, container daemons, or search indexers in isolation.

Mitigating Directory Cache Exhaustion

If dcsnoop reveals persistent cache misses across frequently accessed directories, inspect your kernel VFS cache tuning:

# Check current VFS cache pressure (default is 100)
cat /proc/sys/vm/vfs_cache_pressure

• A high vfs_cache_pressure value (> 100) causes the Linux kernel to reclaim dentry and inode memory aggressively, increasing miss rates.
• Lowering the value to 50 encourages the kernel to retain directory entries in RAM, dramatically reducing disk lookups for heavy filesystem workloads:

  sudo sysctl -w vm.vfs_cache_pressure=50

• To make this setting permanent across reboots, add vm.vfs_cache_pressure = 50 to /etc/sysctl.d/99-vfs-cache.conf.