Standard Linux storage performance monitoring tools such as iostat report block-level metrics averaged across intervals, completely masking millisecond-scale tail latency outliers. In production databases and web servers, individual slow I/O operations—such as an fsync() stalling for 500ms due to jbd2 journal contention or block allocation lock wait—can cascade into widespread thread pool exhaustion. The eBPF-powered ext4slower utility instruments ext4 filesystem operations directly to trace slow reads, writes, and syncs on an event-by-event basis.
How ext4slower Instruments Filesystem Operations
Rather than observing physical disk queues, ext4slower attaches kprobes to native ext4 VFS entry points:
ext4_file_read_iter(): Traces buffered and direct file read requests.ext4_file_write_iter(): Traces write operations, recording time spent waiting on page cache locks or journal transactions.ext4_file_open(): Measures pathname resolution and inode retrieval latency.ext4_sync_file(): Instrumentsfsync()latency down to fractional milliseconds.
Installing the BCC Tracing Toolkit
Ensure the BPF Compiler Collection (BCC) is installed on your Linux distribution:
# Ubuntu / Debian
sudo apt update
sudo apt install -y bpfcc-tools linux-headers-$(uname -r)
# RHEL / Rocky Linux / Fedora
sudo dnf install -y bcc-tools kernel-devel-$(uname -r)
The executable is typically installed at /usr/sbin/ext4slower-bpfcc or /usr/share/bcc/tools/ext4slower.
Tracing Slow ext4 Operations in Real Time
Execute ext4slower with superuser privileges, defining a minimum latency threshold in milliseconds:
# Trace ext4 operations taking longer than 10 milliseconds
sudo ext4slower-bpfcc 10
When an ext4 operation exceeds the specified threshold, ext4slower prints a high-precision event record:
TIME(s) COMM PID T BYTES OFF_KB LAT(ms) FILENAME
15:42:01 mysqld 14820 W 16384 40960 18.42 ibdata1
15:42:05 postgres 22041 S 0 0 85.12 16384
15:42:19 nginx 19820 R 65536 1024 12.04 index.html
15:42:31 app.worker 30105 W 4096 2048 45.18 session.log
The output columns identify:
- TIME(s): Timestamp of the completed operation.
- COMM & PID: The process responsible for the slow call.
- T (Type): Operation type:
R(Read),W(Write),O(Open), orS(Sync). - BYTES: Number of bytes transferred.
- LAT(ms): Elapsed latency in milliseconds.
- FILENAME: Target filename or inode identifier.
Diagnosing Journal Lock Contention (jbd2)
If ext4slower reveals frequent sync stalls on database journal files:
- Inspect active mount options on the target ext4 volume:
mount | grep ext4 - Enable writeback journaling or adjust journal commit intervals to prevent frequent transaction stalls:
# Tune commit interval to 30 seconds for non-critical logging partitions sudo mount -o remount,commit=30 /var/log - Re-run
ext4slower-bpfcc 10to verify that filesystem latency outliers disappear.