How to Trace ext4 Filesystem Latency in Linux Using ext4dist

In high-throughput Linux environments hosting web servers, database clusters, and container workloads, storage latency bottlenecks frequently arise at the filesystem layer rather than the physical disk controller. The fourth extended filesystem (ext4) remains the default standard across enterprise distributions like Ubuntu, Debian, and Red Hat Enterprise Linux. However, lock contention, journal flushing delays, delayed allocation overhead, or heavy synchronous writes (fsync) can cause noticeable request queuing.

Traditional utilities like iostat or vmstat report block-level metrics, failing to capture operations fulfilled entirely by the Linux page cache or stalled by kernel filesystem locks. To analyse ext4 latency distributions with zero sampling bias and negligible overhead, Linux performance engineers use ext4dist. Powered by extended Berkeley Packet Filters (eBPF), ext4dist measures the duration of ext4 read, write, open, and sync operations, summarising them into real-time logarithmic histograms. This tutorial explains how to install, configure, and use ext4dist to diagnose filesystem performance bottlenecks in Linux.

Understanding ext4 Filesystem Latency and ext4dist

When an application reads or writes a file on an ext4 partition, the operating system executes kernel routines inside the ext4 filesystem driver (such as ext4_file_read_iter, ext4_file_write_iter, ext4_file_open, and ext4_sync_file).

Filesystem-level latency includes:

  • Page Cache Lookup: Checking the RAM page cache for preloaded file data.
  • Lock Contention: Waiting on inode or extent allocation mutexes during concurrent writes.
  • Block I/O Dispatch: Issuing requests to disk block queues when reading un-cached blocks or flushing dirty pages.
  • Journal Commits: Flushing transaction blocks to the JBD2 journal during fsync() or synchronous directory updates.

ext4dist attaches eBPF kprobes directly to these ext4 kernel functions. By measuring timestamps between the start and completion of each operation inside the kernel, ext4dist captures microsecond-level latency distributions without flooding user space with massive event streams.

Step 1: Install BCC Tools on Linux

ext4dist is provided as part of the BPF Compiler Collection (BCC) tools suite. Ensure your system runs Linux kernel 4.9 or newer with eBPF support enabled.

On Ubuntu and Debian:

sudo apt update
sudo apt install -y bpfcc-tools linux-headers-$(uname -r)

On Debian and Ubuntu, BCC utilities append the -bpfcc suffix (e.g., ext4dist-bpfcc). To run tools without entering full paths, add /usr/sbin to your shell profile:

echo 'export PATH=$PATH:/usr/sbin' >> ~/.bashrc
source ~/.bashrc

On RHEL, Rocky Linux, and Fedora:

sudo dnf install -y bcc-tools kernel-devel-$(uname -r)
export PATH=$PATH:/usr/share/bcc/tools

On Arch Linux:

sudo pacman -S bcc bcc-tools linux-headers

Step 2: Trace System-Wide ext4 Operation Latency

To record ext4 operations across all running processes and display a summary histogram, execute ext4dist with administrative privileges:

sudo ext4dist-bpfcc

Allow the tool to gather telemetry while disk activity occurs, then press Ctrl + C to end tracing. ext4dist categorises events into distinct operation histograms:

Tracing ext4 operation latency... Hit Ctrl-C to end.
^C

operation = 'read'
     usecs               : count     distribution
         0 -> 1          : 1240     |****************************************|
         2 -> 3          : 312      |**********                              |
         4 -> 7          : 45       |*                                       |
         8 -> 15         : 12       |                                        |
        16 -> 31         : 4        |                                        |
        32 -> 63         : 0        |                                        |

operation = 'write'
     usecs               : count     distribution
         0 -> 1          : 85       |**                                      |
         2 -> 3          : 410      |************                            |
         4 -> 7          : 1320     |****************************************|
         8 -> 15         : 180      |*****                                   |
        16 -> 31         : 34       |*                                       |
        32 -> 63         : 6        |                                        |

In this trace:

  • Reads: The majority completed in 0 to 1 microseconds, demonstrating excellent page cache hit ratios.
  • Writes: Operations clustered around 4 to 7 microseconds, reflecting fast asynchronous buffering into the operating system page cache.

Step 3: Measure Latency in Milliseconds

Under heavy production load or during storage failover testing, microsecond granularity can produce excessively granular distributions. Switch to millisecond reporting using the -m flag:

sudo ext4dist-bpfcc -m
operation = 'sync'
     msecs               : count     distribution
         0 -> 1          : 42       |****************************************|
         2 -> 3          : 15       |**************                          |
         4 -> 7          : 8        |*******                                 |
         8 -> 15         : 3        |**                                      |
        16 -> 31         : 1        |                                        |

Operations in the sync section reflect calls to fsync() and fdatasync(). Rows appearing above 16 ms suggest journal lock contention or queue saturation on the physical disk array.

Step 4: Continuous Periodic Sampling

To monitor filesystem trends continuously without manually halting execution, pass an interval and count parameter:

# Output a fresh latency histogram every 5 seconds for 4 iterations
sudo ext4dist-bpfcc 5 4

This allows administrators to monitor filesystem responsiveness during scheduled database re-indexing, batch archiving, or container deployments.

Step 5: Output Timestamps with Periodic Interval Summaries

When correlating filesystem latency with external monitoring alarms (such as Prometheus or Datadog alerts), add the -T flag to prepend human-readable timestamps to each interval summary:

sudo ext4dist-bpfcc -T 10 3

Sample output:

14:40:12
operation = 'read'
     usecs               : count     distribution
         0 -> 1          : 512      |****************************************|
...

Diagnosing High ext4 Latency

If ext4dist reveals that read or write operations consistently register in the multi-millisecond ranges:

  • High ‘read’ Latency: Applications are frequently missing the Linux page cache, forcing synchronous disk reads. Verify available system RAM and check /proc/sys/vm/vfs_cache_pressure.
  • High ‘sync’ Latency: High sync latency indicates that database transaction commits are delayed waiting for ext4 JBD2 journal writes. Ensure the filesystem is mounted with noatime and consider allocating write-cache-backed storage arrays.
  • Identify Specific Slow Files: If ext4dist detects high latency, run fileslower-bpfcc 10 or ext4slower-bpfcc 10 to capture the specific process names, filenames, and PIDs exceeding 10 milliseconds.

Get the best tech tips delivered straight to your inbox.

Join thousands of readers mastering Apple, Google, Microsoft, and Linux.