How to Trace Slab Memory Allocation Latency in Linux Using eBPF

In high-throughput Linux environments handling millions of network packets, file descriptors, or container tasks, operating system memory performance extends far beyond simple user-space RAM consumption. Inside the Linux kernel, objects like directory entries (dentries), inodes, and socket buffers are allocated via the Slab Allocator (SLUB / SLAB). Under sustained memory pressure or heavy memory fragmentation, kernel allocations via kmalloc() can stall while waiting for direct page reclaim or memory compaction. Tracing slab memory allocation latency using eBPF reveals the exact processes and kernel caches responsible for memory allocation stalls.

The Architecture of the Linux SLUB Allocator

The Linux kernel uses the SLUB allocator to eliminate external memory fragmentation caused by allocating small, fixed-size objects from the primary Buddy Allocator page pool. The SLUB architecture divides memory into three tiers:

• Fast Path (Per-CPU Caching): The kernel retrieves a pre-allocated object directly from the CPU’s local kmem_cache_cpu structure with lockless atomics. This takes less than 10 nanoseconds.
• Medium Path (Per-Node Partial Lists): If the per-CPU slab is exhausted, the allocator acquires the node lock and retrieves slabs from kmem_cache_node partial lists.
• Slow Path (Buddy Allocator Page Allocation): If no partial slabs exist, the allocator calls into the page allocator (alloc_pages()). Under low free-memory conditions, this slow path triggers direct reclaim, dirty page writeback, or memory compaction, causing latency spikes ranging from milliseconds to entire seconds.

Prerequisites and Tooling

Tracing kernel slab routines requires superuser privileges and an eBPF-enabled Linux kernel (5.4 or higher recommended):

# Ubuntu / Debian
sudo apt update && sudo apt install bpftrace bpfcc-tools linux-headers-$(uname -r)

# RHEL / Rocky Linux
sudo dnf install bpftrace bcc-tools kernel-devel-$(uname -r)

Tracing Slow kmalloc Allocations with bpftrace

To pinpoint allocations that experience severe slow-path latency (exceeding 100 microseconds), execute this targeted bpftrace script:

sudo bpftrace -e '
#include <linux/slab.h>

kprobe:kmem_cache_alloc {
    @start[tid] = nsecs;
    @cache[tid] = arg0;
}

kretprobe:kmem_cache_alloc {
    $start = @start[tid];
    if ($start) {
        $lat_us = (nsecs - $start) / 1000;
        if ($lat_us > 100) { // Log allocations taking over 100 microseconds
            $cache = (struct kmem_cache *)@cache[tid];
            printf("%s [SLOW SLAB] Comm: %-16s | Cache: %-20s | Lat: %d us
",
                time("%H:%M:%S"), comm, $cache->name, $lat_us);
        }
        delete(@start[tid]);
        delete(@cache[tid]);
    }
}

END {
    clear(@start);
    clear(@cache);
}'

Whenever a kernel slab allocation encounters latency due to memory compaction or lock contention, the script reports the process, the affected slab cache (e.g., kmalloc-512, dentry, or skbuff_head_cache), and the duration in microseconds.

Inspecting Active Slab Caches with slabtop

To correlate latency events with aggregate memory usage across slab caches, use the interactive slabtop tool:

sudo slabtop -s c

Review the top memory-consuming caches:

• dentry: Directory entry cache. Elevated counts indicate massive directory traversal.
• inode_cache: Inodes held in RAM for open or recently accessed files.
• buffer_head: Filesystem block metadata caching.
• kmalloc-*: General-purpose variable-size allocations requested by device drivers and kernel modules.

Remediation and Kernel Memory Tuning

• Adjusting vm.min_free_kbytes: If allocations frequently stall during direct reclaim, increase the kernel’s minimum free reserve in /etc/sysctl.d/99-memory.conf:

  # Default is often small; increasing to 256MB–512MB provides headroom for burst allocations
  sudo sysctl -w vm.min_free_kbytes=262144

• Tuning Transparent Huge Pages (THP) Defrag: When THP compaction runs synchronously, it introduces severe memory allocation pauses. Configure THP defrag to background mode:

  echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

• Purging Unused Slab Memory: To manually test whether slab caches reclaim cleanly, drop clean caches via echo 2 | sudo tee /proc/sys/vm/drop_caches.