How to Trace Kernel Slab Allocations in Linux Using slabratetop

In high-performance Linux servers, memory allocation bottlenecks frequently originate inside the Linux kernel’s slab allocator rather than in user-space application memory. The kernel uses the slab allocator (SLUB or SLAB) to manage pools of pre-allocated, fixed-size data structures—such as directory entries (dentry), file inodes (inode_cache), network socket buffers (skbuff), and task descriptors (task_struct). While utilities such as slabtop and /proc/slabinfo display the current aggregate memory consumed by each slab cache, they only report cumulative size metrics.

Static slab reports cannot reveal allocation velocity—the rate at which kernel structures are created, recycled, or exhausted over time. When investigating container memory leaks, cache thrashing, or high kernel CPU utilization in kswapd, performance engineers need to trace the rate of slab allocation calls per second. The BPF Compiler Collection (BCC) provides slabratetop, an eBPF utility designed to monitor kernel slab allocation frequency in real time. This guide explains how to trace kernel slab allocations using slabratetop in Linux.

Understanding How slabratetop Operates via eBPF

The slabratetop utility attaches an extended Berkeley Packet Filter (eBPF) kprobe to the kernel’s core slab allocation function: kmem_cache_alloc() and its variants (such as kmem_cache_alloc_node()). Whenever the kernel or any driver requests memory from a designated slab cache, the eBPF probe records:

• The targeted slab cache name (e.g. dentry, ext4_inode_cache, kmalloc-512).
• The calling kernel function or process context.
• The total count and calculated byte volume of allocations completed during the sampling window.

By aggregating counts directly within an in-kernel BPF hash table, slabratetop provides high-resolution allocation frequency tracking without imposing noticeable context-switching overhead.

Prerequisites and Tool Installation

The utility requires Linux kernel 4.9 or newer with BPF enabled, alongside root or CAP_SYS_ADMIN privileges.

Installing on Debian and Ubuntu

1. Update the local APT repository cache:

sudo apt update

2. Install the BCC tools package and Linux kernel headers:

sudo apt install bpfcc-tools linux-headers-$(uname -r)

The executable is available as slabratetop-bpfcc under /usr/sbin/.

Installing on RHEL, Fedora, and Rocky Linux

Install the package via dnf:

sudo dnf install bcc-tools kernel-devel-$(uname -r)

The binary is located at /usr/share/bcc/tools/slabratetop.

Monitoring Kernel Slab Allocation Rates

To launch slabratetop with its default 1-second display refresh interval, execute the command with root privileges:

sudo slabratetop-bpfcc

The terminal clears and prints a live-updating dashboard listing the top slab caches ranked by allocation velocity:

Tracing... Output every 1 secs. Hit Ctrl-C to end.

08:20:14
CACHE                           ALLOCS      BYTES
dentry                           14205    2727360
ext4_inode_cache                  8912   10337920
kmalloc-512                       4510    2309120
skbuff_head_cache                 3120     798720
buffer_head                       1840     191360
task_struct                        120     786432

Interpreting the Dashboard Columns

• CACHE: The specific kernel slab cache receiving allocation requests.
• ALLOCS: The number of discrete allocations fulfilled by this cache during the sampling interval.
• BYTES: The total memory volume allocated during the interval, calculated by multiplying ALLOCS by the cache’s object size.

Configuring Refresh Intervals and Custom Sort Orders

Customise slabratetop‘s display parameters to align with specific diagnostic scenarios:

Adjusting Refresh Intervals and Run Count

To refresh every 5 seconds for a total of 6 cycles (30 seconds total profiling):

sudo slabratetop-bpfcc 5 6

This longer interval smooths out transient noise and captures sustained workload patterns accurately.

Sorting by Allocated Bytes Rather than Allocation Count

By default, slabratetop sorts output rows by ALLOCS (frequency). To rank caches by total memory throughput (BYTES), use the -s or --sort flag:

sudo slabratetop-bpfcc -s bytes

This highlights large data structures (such as task_struct or ext4_inode_cache) that may consume massive memory volumes despite lower invocation counts.

Diagnostic Insights and Root Cause Analysis

Monitoring allocation velocity reveals specific operational bottlenecks:

• High dentry and inode Allocations: If dentry and filesystem inode caches dominate the top spots with tens of thousands of allocations per second, a background process (such as a recursive find, file indexer, or container scanner) is rapidly traversing deep directory trees.
• High skbuff_head_cache Allocation: Spikes in network socket buffers indicate high-packet network throughput, micro-bursts, or unhandled inbound network traffic floods requiring kernel socket buffer expansion.
• Buffer Cache Churn: Heavy buffer_head allocations point to direct raw disk metadata operations, database index rebuilds, or unbuffered block device scanning.

Operational Safeguards

• Production-Safe eBPF Profiling: Because slabratetop increments in-kernel hash table counters and only dumps top-N entries to user space at the specified interval, CPU overhead remains negligible even during heavy memory allocation spikes.
• Kernel Memory Reclaim: Remember that slab allocations reflect memory grants. Modern Linux kernels automatically reclaim inactive slab caches (like dentry) when user applications require anonymous memory, provided vm.vfs_cache_pressure is configured properly.