How to Trace Directory-Level File System Operations in Linux Using dirtop

When storage I/O performance degrades on a production Linux server, identifying which processes and directories are generating excessive operations is critical for rapid remediation. While device-level tools like iostat measure physical disk saturation, they cannot identify which specific directories on disk are absorbing operations. The dirtop utility, part of the BPF Compiler Collection (BCC), dynamically traces directory-level file system operations in real time using extended Berkeley Packet Filters (eBPF), streaming top directories ranked by activity.

How dirtop Observes Directory Operations

In high-throughput environments, applications such as search indexing daemons, continuous integration runners, or un-pruned container caches frequently create, delete, and read thousands of files within specific directory trees. Traditional tools like lsof or iotop report per-process statistics, but they cannot aggregate operations hierarchically by folder path.

The dirtop utility hooks into kernel VFS and file system callbacks, dynamically extracting file descriptor parent paths to aggregate:

• Total read and write invocations per directory.
• Throughput volume in kilobytes.
• Real-time ranking of the most heavily accessed directories system-wide.

Installing the BCC Tools Suite

To run dirtop, install the BCC package and kernel development headers corresponding to your active kernel version.

On Debian, Ubuntu, and Linux Mint:

sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

On RHEL, Rocky Linux, AlmaLinux, or Fedora:

sudo dnf install bcc-tools kernel-devel-$(uname -r)

The executable is typically installed at /usr/sbin/dirtop-bpfcc or /usr/share/bcc/tools/dirtop.

Streaming Top Directory Activity in Real Time

To launch dirtop with default sampling, execute the command with root privileges:

sudo dirtop-bpfcc

The utility clears the terminal and streams an interactive top-style table updating every second:

Tracing... Output every 1 secs. Hit Ctrl-C to end.

READS  WRITES R_Kb    W_Kb    DIRECTORY
1420   380    5680    1520    /var/lib/postgresql/data
840    120    3360    480     /var/log/nginx
420    0      1680    0       /usr/share/zoneinfo
0      140    0       560     /tmp/cache

Each column details directory performance metrics:

• READS / WRITES: Total read and write system call invocations during the sample window.
• R_Kb / W_Kb: Cumulative read and write data transferred in kilobytes.
• DIRECTORY: The resolved parent directory path where the operations occurred.

Customising Refresh Intervals and Sample Snapshots

To prevent continuous terminal clearing when redirecting output into logging scripts or diagnostic bundles:

# Sample every 5 seconds for 12 iterations without clearing the terminal screen
sudo dirtop-bpfcc -C 5 12 > /var/log/dirtop_audit.log

The -C flag preserves scrollback history, making it simple to capture sequential snapshots during a database load test or disk thrashing incident.

Sorting Directory Telemetry by Throughput Metrics

Depending on the nature of your storage bottleneck, sort directories by specific performance vectors:

• Sort by Read Throughput: To identify which folders are causing heavy cache misses and cold disk reads:

  sudo dirtop-bpfcc -s r

• Sort by Write Throughput: To isolate which application directories are consuming write bandwidth:

  sudo dirtop-bpfcc -s w

Remediating Directory-Level Bottlenecks

When dirtop exposes directory hot-spots, consider the following architectural adjustments:

• Mounting Hot Temporary Directories on tmpfs: If directories like /tmp/cache or local build workspaces dominate write activity, mount them into RAM using tmpfs to eliminate disk write latency.
• Adjusting Mount Options: Apply noatime or nodiratime in /etc/fstab to prevent the kernel from writing timestamp updates whenever files in busy directories are accessed.
• Relocating Database Clusters: Move heavily trafficked data directories (e.g., /var/lib/mysql) onto dedicated NVMe storage volumes.

Using dirtop provides immediate, directory-level clarity into file system workloads, enabling system engineers to target optimisation efforts with precision.