How to Trace Linux Software RAID Flush Events and Latency Using mdflush

In high-availability Linux storage architectures, the Multiple Device (md) driver provides software RAID capabilities (such as RAID 0, 1, 5, 6, and 10), aggregating individual NVMe SSDs or SATA hard drives into resilient logical volumes. However, to guarantee data persistence during power outages or sudden shutdowns, filesystems periodically issue barrier and flush commands that force the md driver to flush dirty cache pages across all underlying physical member disks. When individual member drives experience intermittent controller delays or rebuild contention, flush latency can spike, freezing database write transactions. The mdflush tool from the BCC (BPF Compiler Collection) suite intercepts kernel software RAID flush events in real time, measuring flush latency and identifying storage bottlenecks directly in your terminal.

Understanding How mdflush Profiles Software RAID

When an application executes a synchronous write or an fsync system call, the filesystem sends a flush bio (a block I/O request with the REQ_PREFLUSH or REQ_FUA flag) down to the md driver. The software RAID driver must propagate this flush request to every active disk in the array and wait for all physical devices to acknowledge that internal disk hardware caches have been written to non-volatile storage.

Using eBPF kprobes, mdflush attaches to internal kernel entry points within the Multiple Device driver (specifically md_flush_request). It records when a flush request enters the software RAID layer, tracks the completion event when all underlying member drives complete their cache flushes, and computes the round-trip latency in milliseconds. Much like evaluating filesystem sync delays with syncsnoop or tracking per-event storage transactions using biosnoop, mdflush provides focused visibility into software array synchronization overhead.

Installing Prerequisites and BCC Packages

To run mdflush, your system requires kernel headers matching your running Linux distribution and the BCC tools package. On Debian and Ubuntu systems, install the dependencies via apt:

sudo apt update && sudo apt install -y bpfcc-tools linux-headers-$(uname -r)

On Red Hat Enterprise Linux, Rocky Linux, or Fedora, use dnf:

sudo dnf install -y bcc-tools kernel-devel-$(uname -r)

The executable is typically installed at /usr/sbin/mdflush or /usr/sbin/mdflush-bpfcc. Because eBPF programs hook into internal kernel subsystems, executing mdflush requires sudo or root administrative rights.

Tracing Live Software RAID Flush Events

To trace flush events across all active software RAID devices on your Linux host, launch mdflush in your terminal:

sudo mdflush

The tool outputs an event row every time a flush request completes against an md array:

Tracing md flush events... Hit Ctrl-C to end.
TIME(s)            PID     COMM             DEVICE
0.000000000        842     jbd2/md0-8       md0
0.104289102        1420    postgres         md1
2.451829014        920     kworker/u16:2    md0
5.819201948        1420    postgres         md1

The output columns provide crucial diagnostics:

• TIME(s): Time elapsed in fractional seconds since the monitoring session began.
• PID: The process ID that initiated or queued the flush request.
• COMM: Executable command name (such as the ext4 journal daemon jbd2, database engines like postgres, or kernel worker threads).
• DEVICE: The specific software RAID logical device node (e.g. md0, md1) servicing the flush.

Adding Wall-Clock Timestamps for Incident Correlation

When diagnosing intermittent application timeouts, correlating flush storms with enterprise monitoring alerts requires absolute wall-clock time rather than elapsed session seconds. Use the -t or -T parameter:

sudo mdflush -T

With -T enabled, timestamps appear in standard HH:MM:SS format (for example, 15:42:18), allowing system administrators to match high flush frequencies directly against database transaction commit logs or web server timeout spikes.

Diagnosing RAID Array Contention and Latency Spikes

A software RAID array is only as fast as its slowest member disk during a cache flush operation. If one mechanical drive or SSD in a RAID 1 or RAID 5 array is degraded, failing, or suffering from bad sector retries, every single flush operation stalls until that troubled drive responds.

When you detect excessive flush activity on an md device, pair mdflush with companion performance tools to isolate the degraded member:

• Inspect overall storage latency distributions with biolatency to verify whether underlying member drives (e.g. sda, sdb) exhibit tail latencies.
• Check whether memory pressure is forcing dirty page flushes using cachetop.
• Query RAID array health status using cat /proc/mdstat or mdadm --detail /dev/md0 to confirm whether a background resync or rebuild operation is competing for storage bandwidth.

Press Ctrl + C at any time to detach the eBPF kprobes and safely conclude the monitoring session.