How to Trace Slow Synchronous Filesystem I/O in Linux Using fileslower

Storage subsystem latency spikes can cause database transactions to stall, web application response times to degrade, and background batch processing jobs to exceed scheduled execution windows. While traditional Linux monitoring tools such as iostat, sar, and vmstat report aggregated storage device utilization and average wait times, they cannot identify which specific processes, files, or thread operations are encountering extreme filesystem latency. The fileslower tool, part of the BPF Compiler Collection (BCC), solves this diagnostic visibility gap by attaching eBPF kernel probes to common Virtual File System (VFS) and filesystem read/write routines across ext4, XFS, Btrfs, and ZFS. It captures synchronous file operations that exceed a configurable latency threshold in real time. This article explains how to install BCC tools, trace slow filesystem operations, and isolate storage bottlenecks in Linux using fileslower.

Understanding Filesystem Latency and eBPF Tracing with fileslower

In Linux, when an application invokes synchronous file I/O system calls such as read(), write(), pread64(), or pwrite64(), the execution thread blocks until the filesystem completes the request or returns cached data. If the requested data is resident in the page cache, latency is measured in nanoseconds. However, if a cache miss forces mechanical head movement on hard drives, block allocation contention on NVMe SSDs, or journal lock waiting on filesystems, latency can suddenly jump into tens or hundreds of milliseconds.

Traditional tools like strace can measure system call duration, but running strace on high-throughput production servers incurs severe performance overhead. fileslower uses eBPF probes attached to specific filesystem kernel drivers (including ext4_file_read_iter, xfs_file_buffered_write, and equivalent routines):

• Records the timestamp and parameters when a synchronous file operation begins.
• Calculates the elapsed duration upon function return.
• Compares the duration against your specified millisecond threshold. If the operation was faster than the threshold, the data is discarded instantly in kernel memory without entering user space.
• If the operation exceeded the threshold, fileslower extracts the process name, Process ID (PID), I/O size, operation type, and full file path.

Installing BCC Tools on Ubuntu, Debian, and Enterprise Linux

To use fileslower, ensure the BCC toolchain and kernel headers matching your running kernel are installed. On Debian and Ubuntu systems, execute:

sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

On Red Hat Enterprise Linux, AlmaLinux, Rocky Linux, and Fedora systems, run:

sudo dnf install bcc-tools kernel-devel-$(uname -r)

On Debian and Ubuntu, BCC utilities include a -bpfcc suffix (/usr/sbin/fileslower-bpfcc), whereas Enterprise Linux distributions locate them in /usr/share/bcc/tools/fileslower.

Tracing Slow Filesystem Operations Exceeding Latency Thresholds

To begin tracing slow file I/O across the entire system, execute fileslower with superuser privileges, specifying a minimum latency threshold in milliseconds. For example, to report any file read or write that takes longer than 10 milliseconds, run:

sudo fileslower-bpfcc 10

When slow operations occur, fileslower outputs real-time event rows structured as follows:

TIME(s)  COMM             PID    D BYTES   LAT(ms) FILENAME
14:35:02 postgres         4812   W 16384     45.21 base/16384/24810
14:35:04 mysqld           3102   R 8192      22.14 ibdata1
14:35:07 rsync            8921   R 131072    88.45 backup_archive.tar.gz
14:35:10 redis-server     2104   W 4096      15.02 dump.rdb

Understanding each telemetry column enables precise root-cause analysis:

• TIME(s): Time of day or elapsed trace duration.
• COMM: The command/process name responsible for initiating the slow I/O operation.
• PID: The Linux process identifier.
• D (Direction): The operation type—R for read operations or W for write operations.
• BYTES: The size of the read or write request in bytes (e.g. 16,384 bytes = 16 KB).
• LAT(ms): The exact duration of the blocking operation in milliseconds.
• FILENAME: The target file being accessed.

Targeting Specific PIDs and Filtering Operational Scope

To isolate storage latency affecting a critical database or container workload without monitoring irrelevant background system activity, filter by process ID:

1. Trace a Specific Process ID (-p): To capture slow I/O transactions exceeding 5 ms specifically for a PostgreSQL or MySQL server running under PID 4812, execute:

sudo fileslower-bpfcc -p 4812 5

2. Distinguishing Reads from Writes: While fileslower traces both reads and writes by default, observing write latency (W) typically indicates journal lock contention, disk queue saturation, or dirty page flushing delays, whereas read latency (R) highlights cold page cache misses or fragmented disk blocks.

3. Low-Latency High-Frequency Profiling: On high-performance enterprise all-flash NVMe arrays, standard file operations complete in under 1 millisecond. To trace microsecond-level degradation, pass fractional millisecond thresholds (for example, 1 for 1 ms, or 0.5 for 500 microseconds):

sudo fileslower-bpfcc 1

Diagnosing and Remediating Filesystem Bottlenecks

When fileslower exposes repeated latency spikes across specific processes or files, investigate the following system factors:

1. Storage Queue Congestion: Cross-reference fileslower results with biolatency-bpfcc or iostat -xz 1. If device utilization approaches 100% and average queue lengths spike alongside fileslower events, underlying storage bandwidth is exhausted.

2. Filesystem Journaling Contention: Repeated write stalls on ext4 or XFS metadata files often occur when synchronous journaling (data=journal or heavy fsync() calls) forces serialization. Consider adjusting mount options (such as noatime) or placing Write-Ahead Logs (WAL) on dedicated high-speed NVMe storage.

3. Page Cache Starvation: If read operations dominate slow entries on files that should be memory-resident, inspect system memory pressure using cachestat-bpfcc and consider increasing RAM allocations or pinning critical working sets.

By leveraging fileslower, Linux systems engineers and database administrators gain granular, per-file latency visibility, allowing rapid isolation of storage bottlenecks with negligible overhead.