How to Trace Direct Memory Reclaim Latency in Linux Using drsnoop

Under heavy memory pressure, when the asynchronous kernel swap daemon (kswapd) fails to replenish free page lists fast enough to keep up with process allocations, executing threads are forced into direct memory reclaim. During direct reclaim, user-space application threads stall synchronously while scanning page lists, evicting cache entries, and writing dirty pages to disk. The eBPF-powered drsnoop tool instruments the kernel memory management subsystem to record allocation latency down to fractional milliseconds.

Understanding Direct Reclaim Latency and Application Stalls

Direct reclaim introduces severe tail-latency spikes in production microservices and database engines:

• Synchronous Blocking: When memory watermarks breach low thresholds, allocation routines in __alloc_pages_direct_reclaim() force requesting threads into wait states.
• I/O Bottlenecks: If direct reclaim attempts to free dirty file-backed pages, threads can block indefinitely waiting for storage writebacks to flush.
• Kernel Watermark Levels: Tuning vm.min_free_kbytes and vm.watermark_scale_factor determines how aggressively kswapd initiates background reclaim before direct reclaim triggers.

Installing the BCC Tracing Toolkit

The drsnoop utility is bundled with the BPF Compiler Collection (BCC):

# Debian / Ubuntu
sudo apt update
sudo apt install -y bpfcc-tools linux-headers-$(uname -r)

# RHEL / Rocky Linux / Fedora
sudo dnf install -y bcc-tools kernel-devel-$(uname -r)

The binary is located in /usr/sbin/drsnoop-bpfcc or /usr/share/bcc/tools/drsnoop.

Tracing Live Direct Reclaim Events

Execute drsnoop with superuser privileges to begin monitoring reclaim latency across all active processes:

sudo drsnoop-bpfcc

When processes encounter allocation stalls, drsnoop outputs real-time event telemetry:

TIME(s)            COMM             PID    LAT(ms) PAGES
12:15:32.418291    mysqld           14201  18.42   128
12:15:33.109482    java             22810  45.18   512
12:15:35.882190    redis-server     8902   8.12    64

The telemetry columns indicate:

• TIME(s): High-precision timestamp of the completed reclaim operation.
• COMM & PID: The process name and process ID stalled in direct reclaim.
• LAT(ms): Duration in milliseconds that the application thread was blocked.
• PAGES: Number of 4KB memory pages successfully reclaimed during the stall.

Filtering by Minimum Latency Threshold

To isolate critical latency outliers and ignore minor sub-millisecond reclaims:

# Trace only direct reclaim events taking longer than 10 milliseconds
sudo drsnoop-bpfcc -m 10

Tuning Kernel Memory Watermarks to Prevent Direct Reclaim

If drsnoop reveals frequent stalls, adjust kernel reclaim thresholds to make background kswapd run earlier and more aggressively:

1. Inspect current kernel watermark settings:

   sysctl vm.min_free_kbytes
   sysctl vm.watermark_scale_factor

2. Increase watermark scale factor to expand the margin between minimum and high memory thresholds:

   # Double default watermark scale factor (default is 10)
   sudo sysctl -w vm.watermark_scale_factor=200
   
   # Increase minimum free reserve to prevent allocation stalls under bursty traffic
   sudo sysctl -w vm.min_free_kbytes=131072

3. Persist the configuration across reboots in /etc/sysctl.d/99-memory-tuning.conf:

   vm.watermark_scale_factor = 200
   vm.min_free_kbytes = 131072

4. Re-run drsnoop-bpfcc during peak workloads to confirm that direct reclaim frequency drops to zero.