How to Trace Memory Compaction Stall Latency in Linux Using compactsnoop

High-performance Linux servers powering in-memory caches (such as Redis or Memcached), virtual machine hypervisors, and large database engines rely on contiguous physical memory pages to satisfy Transparent Huge Pages (THP) allocations. However, as a server experiences uptime and handles millions of varying memory allocations, physical RAM becomes severely fragmented. When an application requests an order-9 contiguous 2MB memory block, the Linux kernel must pause execution to perform synchronous memory compaction—shifting active pages around physical memory to assemble contiguous page frames. Tracing compaction stall latency using eBPF exposes exactly when memory defragmentation delays degrade application responsiveness.

The Mechanics of Kernel Memory Compaction

The Linux physical memory Buddy Allocator organizes RAM into power-of-two page orders (from Order-0 for 4KB pages to Order-10 for 4MB contiguous blocks). Over time, small long-lived allocations scatter across memory zones:

• When an application invokes madvise(MADV_HUGEPAGE) or allocates large heaps, the kernel attempts to allocate a contiguous 2MB huge page.
• If no contiguous Order-9 block exists—even if gigabytes of aggregate memory remain free—the kernel triggers Direct Compaction via try_to_compact_pages().
• The kernel scanner walks memory zones from the beginning looking for movable pages, and from the end looking for free page frames, migrating pages sequentially to create contiguous clusters.
• During synchronous compaction, application worker threads block, resulting in sudden latency spikes ranging from 10 milliseconds to multiple seconds.

Prerequisites and Tool Installation

Tracing memory compaction requires root privileges and a Linux kernel with eBPF support (kernel 4.9 or higher):

# Ubuntu / Debian
sudo apt update && sudo apt install bpfcc-tools linux-headers-$(uname -r)

# RHEL / Rocky Linux / Fedora
sudo dnf install bcc-tools kernel-devel-$(uname -r)

On Debian and Ubuntu systems, the executable is available under /usr/sbin/compactsnoop-bpfcc.

Tracing Memory Compaction Events in Real Time

To capture direct memory compaction events across your server, launch compactsnoop with superuser privileges:

sudo compactsnoop-bpfcc

Whenever a kernel compaction cycle executes, the tool streams tabular diagnostic rows:

TIME(s)  COMM             PID    ORDER  LAT(ms)
06:24:12 mongod           1842   9      42.18
06:24:15 redis-server     2910   9      18.45
06:24:22 qemu-system-x86  4102   9      124.80

The column headers indicate:

• TIME(s): Timestamp of the compaction event.
• COMM: Process name of the thread that was paused during compaction.
• PID: Process identifier of the blocked task.
• ORDER: The requested memory page order (e.g., 9 represents a 2MB huge page: 4KB × 2⁹).
• LAT(ms): The duration in milliseconds that the thread was stalled while the kernel rearranged physical memory frames.

Advanced Filtering and Histogram Generation

To measure the latency distribution of compaction delays over extended monitoring intervals, pair compactsnoop with bpftrace:

sudo bpftrace -e '
#include <linux/compaction.h>

kprobe:try_to_compact_pages {
    @start[tid] = nsecs;
    @order[tid] = arg1;
}

kretprobe:try_to_compact_pages {
    $start = @start[tid];
    if ($start) {
        $lat_ms = (nsecs - $start) / 1000000;
        @compaction_lat_ms = hist($lat_ms);
        delete(@start[tid]);
        delete(@order[tid]);
    }
}

END {
    clear(@start);
    clear(@order);
}'

When you terminate the script with Ctrl + C, bpftrace renders a power-of-two histogram showing how many compaction events completed within specific millisecond thresholds.

Diagnostic Interpretations and Kernel Tuning

• Frequent Order-9 Stalls: Confirms that Transparent Huge Pages are fragmenting physical memory. Review current THP defragmentation policies:

  cat /sys/kernel/mm/transparent_hugepage/defrag

• Switching Defrag to Background Mode: If defrag is set to always, application threads block synchronously on allocation. Switch to madvise or defer to delegate compaction to the background kcompactd kernel thread:

  echo defer | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

• Pre-Allocating Static Hugepages: For mission-critical databases, bypass runtime compaction entirely by pre-allocating static 2MB hugepages at boot time via vm.nr_hugepages in /etc/sysctl.d/99-hugepages.conf.