How to Trace XFS Filesystem Latency Outliers in Linux Using xfsslower

XFS is a high-performance, 64-bit journaling filesystem widely deployed across enterprise Linux distributions such as Red Hat Enterprise Linux, Rocky Linux, CentOS Stream, and Ubuntu Server. It excels at managing petabyte-scale storage, parallel input/output operations, and multithreaded database workloads. However, when storage controllers experience queue saturation, disk lockups, or metadata contention during journal flushing, filesystem operations can stall. The BCC (BPF Compiler Collection) tool xfsslower leverages extended Berkeley Packet Filters (eBPF) to trace filesystem calls and report individual operations that breach latency thresholds.

Understanding How xfsslower Measures Filesystem Latency

Standard storage monitoring utilities like iostat or sar report aggregated device-level throughput and average queue lengths. While useful for high-level capacity planning, device metrics obscure brief micro-bursts of extreme latency—often referred to as tail latency or outliers—where a single database commit or file read stalls for several hundred milliseconds.

xfsslower hooks directly into the Linux VFS (Virtual Filesystem Switch) and XFS driver entry points, measuring the precise elapsed time between when an application requests an I/O operation (such as read, write, open, or fsync) and when the XFS filesystem layer completes the request. Because it uses eBPF in-kernel filtering, xfsslower incurs negligible overhead, evaluating millions of fast events in kernel space and emitting only the slow operations to user space.

Installing the BCC Tools Suite

To use xfsslower, install the BPF Compiler Collection package for your Linux distribution:

# Ubuntu / Debian
sudo apt-get update
sudo apt-get install -y bpfcc-tools linux-headers-$(uname -r)

# RHEL 9 / Rocky Linux 9 / AlmaLinux
sudo dnf install -y bcc-tools kernel-devel-$(uname -r)

# Verify execution path
which xfsslower || which /usr/share/bcc/tools/xfsslower

Ensure that your target kernel was compiled with eBPF support enabled (CONFIG_BPF=y and CONFIG_BPF_SYSCALL=y), which is standard on all modern production kernels (version 4.9 and higher).

Running xfsslower with Custom Latency Thresholds

Invoke xfsslower with superuser privileges, specifying a minimum latency threshold in milliseconds. If omitted, the default threshold is 10 ms:

# Trace XFS operations slower than 10 milliseconds
sudo /usr/share/bcc/tools/xfsslower 10

# Capture severe stalls slower than 50 milliseconds
sudo /usr/share/bcc/tools/xfsslower 50

When an operation exceeds the specified threshold, xfsslower logs a real-time event to standard output:

TIME(s)  COMM             PID     T BYTES   OFF_KB   LAT(ms) FILENAME
04:12:08 postgres         18420   S 0       0        84.12   00000001000000000000002A
04:12:15 mariadbd         29112   W 16384   1048576  62.45   ibdata1
04:12:19 rsyslogd         1120    W 4096    256      18.91   syslog
04:12:33 tar              41088   R 131072  524288   112.30  backup.tar.gz

Interpreting Output Columns and Operation Types

Each output line reveals essential diagnostic telemetry:

• TIME(s): The local system timestamp when the operation completed.
• COMM: The executable command name that initiated the filesystem call.
• PID: The process identifier responsible for the request.
• T (Type): The operation performed:
  • R: File read (via xfs_file_read_iter).
  • W: File write (via xfs_file_write_iter).
  • O: File open (via xfs_file_open).
  • S: Synchronous flush (via xfs_file_fsync).
• BYTES: The size of the data payload in bytes.
• OFF_KB: The offset within the file where the operation began, expressed in kilobytes.
• LAT(ms): The actual elapsed duration of the operation in milliseconds.
• FILENAME: The target file path or filename accessed.

Distinguishing Filesystem Bottlenecks from Disk Saturation

When xfsslower flags frequent latency spikes, determine whether the root cause stems from physical drive exhaustion or internal XFS lock contention:

1. Check synchronous flush delays (S): If fsync calls take hundreds of milliseconds while standard reads and writes remain fast, the drive’s volatile cache flush or battery-backed write cache may be overwhelmed by commit frequencies.
2. Correlate with biolatency: Run the BCC tool biolatency simultaneously. If block I/O latency matches the high values seen in xfsslower, the bottleneck lies at the hardware or SAN storage layer. If block I/O is fast (under 2 ms) but xfsslower is slow, XFS metadata lock contention or fragmentation is the culprit.
3. Mitigate allocation stalls: For high-concurrency databases, format filesystems with multiple allocation groups (mkfs.xfs -d agcount=64) and mount with the noatime and logbufs=8 flags to minimise journal commit contention.

By capturing granular I/O latency outliers with xfsslower, systems engineers can identify anomalous database transactions and filesystem stalls before they cascade into broad application timeouts.