How to Trace ext4 Filesystem Latency Outliers in Linux Using ext4slower

Standard Linux storage performance monitoring tools such as iostat report block-level metrics averaged across intervals, completely masking millisecond-scale tail latency outliers. In production databases and web servers, individual slow I/O operations—such as an fsync() stalling for 500ms due to jbd2 journal contention or block allocation lock wait—can cascade into widespread thread pool exhaustion. The eBPF-powered ext4slower utility instruments ext4 filesystem operations directly to trace slow reads, writes, and syncs on an event-by-event basis.

How ext4slower Instruments Filesystem Operations

Rather than observing physical disk queues, ext4slower attaches kprobes to native ext4 VFS entry points:

  • ext4_file_read_iter(): Traces buffered and direct file read requests.
  • ext4_file_write_iter(): Traces write operations, recording time spent waiting on page cache locks or journal transactions.
  • ext4_file_open(): Measures pathname resolution and inode retrieval latency.
  • ext4_sync_file(): Instruments fsync() latency down to fractional milliseconds.

Installing the BCC Tracing Toolkit

Ensure the BPF Compiler Collection (BCC) is installed on your Linux distribution:

# Ubuntu / Debian
sudo apt update
sudo apt install -y bpfcc-tools linux-headers-$(uname -r)

# RHEL / Rocky Linux / Fedora
sudo dnf install -y bcc-tools kernel-devel-$(uname -r)

The executable is typically installed at /usr/sbin/ext4slower-bpfcc or /usr/share/bcc/tools/ext4slower.

Tracing Slow ext4 Operations in Real Time

Execute ext4slower with superuser privileges, defining a minimum latency threshold in milliseconds:

# Trace ext4 operations taking longer than 10 milliseconds
sudo ext4slower-bpfcc 10

When an ext4 operation exceeds the specified threshold, ext4slower prints a high-precision event record:

TIME(s)     COMM       PID    T BYTES   OFF_KB   LAT(ms) FILENAME
15:42:01    mysqld     14820  W 16384   40960      18.42 ibdata1
15:42:05    postgres   22041  S 0       0          85.12 16384
15:42:19    nginx      19820  R 65536   1024       12.04 index.html
15:42:31    app.worker 30105  W 4096    2048       45.18 session.log

The output columns identify:

  • TIME(s): Timestamp of the completed operation.
  • COMM & PID: The process responsible for the slow call.
  • T (Type): Operation type: R (Read), W (Write), O (Open), or S (Sync).
  • BYTES: Number of bytes transferred.
  • LAT(ms): Elapsed latency in milliseconds.
  • FILENAME: Target filename or inode identifier.

Diagnosing Journal Lock Contention (jbd2)

If ext4slower reveals frequent sync stalls on database journal files:

  1. Inspect active mount options on the target ext4 volume:
    mount | grep ext4
  2. Enable writeback journaling or adjust journal commit intervals to prevent frequent transaction stalls:
    # Tune commit interval to 30 seconds for non-critical logging partitions
    sudo mount -o remount,commit=30 /var/log
  3. Re-run ext4slower-bpfcc 10 to verify that filesystem latency outliers disappear.

Get the best tech tips delivered straight to your inbox.

Join thousands of readers mastering Apple, Google, Microsoft, and Linux.