How to Trace Dirty Page Writeback Latency in Linux Using eBPF

High-throughput database engines, message queues, and logging daemons on Linux write data asynchronously into the kernel page cache to achieve near-instantaneous write response times. Periodically, kernel background flush threads (kworker/flush-*) flush these modified “dirty” pages out to underlying NVMe, SSD, or network storage volumes. However, during heavy write bursts or storage hardware saturation, processes can experience severe I/O stalls when forced into synchronous dirty page throttling (the balance dirty pages state). Tracing dirty page writeback latency using eBPF exposes exactly when, why, and for how long system processes are throttled by disk write queues.

Understanding Linux Dirty Page Writeback Architecture

When an application executes the write() system call, data is copied into RAM pages marked as dirty in the page cache. The kernel monitors dirty memory thresholds via two primary parameters:

• vm.dirty_background_ratio / dirty_background_bytes: The threshold at which kernel background threads wake up to write dirty pages to disk asynchronously.
• vm.dirty_ratio / dirty_bytes: The hard ceiling where the kernel pauses application threads and forces them into synchronous writeback mode (balance_dirty_pages()) until dirty memory drops below the threshold.

When an application enters balance_dirty_pages(), its worker threads block, causing severe transaction latency spikes. Tracing these events reveals whether slow disk I/O or aggressive buffering is degrading application performance.

Prerequisites and Environment Setup

Tracing kernel writeback functions requires root privileges and a kernel supporting BPF Type Format (BTF) (Linux 5.4 or higher recommended):

# Ubuntu / Debian
sudo apt update && sudo apt install bpftrace linux-headers-$(uname -r)

# RHEL / Rocky Linux
sudo dnf install bpftrace bpftrace-tools kernel-devel-$(uname -r)

Tracing balance_dirty_pages Throttling Latency

To measure the duration that user-space processes spend blocked waiting for memory dirty pages to flush to storage, deploy the following bpftrace script:

sudo bpftrace -e '
kprobe:balance_dirty_pages {
    @start[tid] = nsecs;
}

kretprobe:balance_dirty_pages {
    $start = @start[tid];
    if ($start) {
        $duration_ms = (nsecs - $start) / 1000000;
        if ($duration_ms > 5) {
            printf("%s [THROTTLE] Comm: %-16s | PID: %-6d | Blocked: %d ms
",
                time("%H:%M:%S"), comm, pid, $duration_ms);
        }
        delete(@start[tid]);
    }
}

END {
    clear(@start);
}'

Any time an application thread is forced to stall for more than 5 milliseconds while dirty pages are flushed to disk, the script logs the process name, PID, and exact blocking duration.

Inspecting Kernel Flush Worker Activity

To verify which storage devices are actively receiving writeback flushes, trace the kernel’s wb_writeback() routine:

sudo bpftrace -e '
tracepoint:writeback:writeback_start {
    printf("%s [WB_START] BDI: %s | Work: %lu pages
",
        time("%H:%M:%S"), str(args->name), args->nr_pages);
}

tracepoint:writeback:writeback_written {
    printf("%s [WB_WRITTEN] BDI: %s | Wrote: %lu pages
",
        time("%H:%M:%S"), str(args->name), args->nr_pages);
}'

This tracepoint monitors the block device interface (BDI) and reports the volume of memory pages written during each flush iteration.

Diagnostic Analysis and Kernel Tuning

• Frequent Balance Dirty Pages Warnings: Indicates that disk write throughput cannot keep pace with application ingestion rates. If dirty memory spikes abruptly, storage controller caches may be saturated.
• Lowering Background Flush Thresholds: Start background flushes earlier by configuring byte-based limits in /etc/sysctl.d/99-dirty.conf:

  # Trigger background flush when dirty memory exceeds 256MB
  vm.dirty_background_bytes = 268435456
  
  # Force synchronous throttling only if dirty memory reaches 1GB
  vm.dirty_bytes = 1073741824

• Aligning Storage Queue Depths: Verify physical drive latency using iostat -xz 1. If %util is 100% and await is high, consider migrating high-write tables to faster NVMe tiers or striped arrays.