In high-throughput Linux environments—such as high-frequency trading servers, heavy database hosts, and Kubernetes nodes handling thousands of container lifecycles—kernel memory management plays a pivotal role in overall system stability. The Linux kernel allocates memory for internal structures (such as inode buffers, socket control blocks, filesystem dentries, and process task descriptors) using the SLUB or SLAB memory allocator, backed by the low-level Buddy page allocator.
When system performance degrades under memory pressure, tools like free -m or standard user-space profilers only report coarse system-wide RAM usage without explaining why kernel memory consumption is expanding. The Linux performance subsystem provides perf kmem, a specialized sub-command dedicated to tracing, recording, and analysing kernel slab object requests, cross-CPU allocations, page frame usage, and allocator latency directly from kernel tracepoints.
Architecture of Kernel Slab and Page Allocation Tracing
The perf kmem tool hooks directly into static kernel tracepoints defined within the kmem subsystem. These tracepoints include:
kmem:kmallocandkmem:kfree: Fired whenever kernel code allocates or frees a generic memory buffer viakmalloc().kmem:kmem_cache_allocandkmem:kmem_cache_free: Fired when dedicated kernel slab caches (such asdentry,ext4_inode_cache, orskbuff_head_cache) allocate or release objects.kmem:mm_page_allocandkmem:mm_page_free: Fired when the kernel Buddy allocator allocates or returns raw 4KB physical page frames.
By capturing these events into an event stream, perf kmem calculates allocation frequency, cumulative bytes requested, cross-node NUMA transfers, and call-site hit rates.
Prerequisites and Kernel Configuration
Profiling kernel memory requires root privileges and access to tracefs. Ensure your Linux distribution has the performance monitoring package installed:
sudo apt update && sudo apt install linux-tools-common linux-tools-$(uname -r)
Ensure that kernel tracepoints are accessible by verifying the tracing debug filesystem:
sudo ls /sys/kernel/debug/tracing/events/kmem/
If tracepoints are not mounted, mount the debug filesystem:
sudo mount -t debugfs none /sys/kernel/debug
Recording Kernel Memory Events with perf kmem record
Before analysing memory allocation metrics, capture live kernel events using perf kmem record.
To record slab allocator events across all CPU cores for 10 seconds, execute:
sudo perf kmem record --slab -- sleep 10
To record physical page allocations (Buddy allocator events) rather than slab caches, use the --page flag:
sudo perf kmem record --page -- sleep 10
To capture both slab cache allocations and raw page frame operations simultaneously while running a workload binary:
sudo perf kmem record --slab --page -- ./workload_script.sh
This writes raw memory trace events into perf.data in your active working directory.
Analysing Slab Allocator Telemetry with perf kmem stat
Once event sampling concludes, use perf kmem stat to parse the data.
To view slab cache statistics sorted by call site allocation volume, execute:
sudo perf kmem --slab stat --caller
A representative output block displays:
------------------------------------------------------------------------------------------------------
Callsite | Total_alloc/Per | Hit | Ping-pong | Frag
------------------------------------------------------------------------------------------------------
__alloc_skb+0x8a | 12451840/512 | 24320 | 210 | 0.00%
kmem_cache_alloc+0xb2 | 4194304/256 | 16384 | 45 | 0.00%
do_sys_openat2+0xe4 | 2097152/128 | 16384 | 12 | 12.50%
Key metrics in the output include:
Callsite: The internal kernel function address and offset responsible for requesting the memory allocation.Total_alloc/Per: The total bytes allocated by this call site followed by the average allocation size in bytes per call.Hit: The total count of successful allocations fulfilled from this call site.Ping-pong: Allocations that were freed on a different CPU core than the one that allocated them. High ping-pong counts indicate cache-line bouncing and lock contention across CPU caches.Frag(Fragmentation): The percentage of wasted memory caused by requesting buffer sizes that do not align with natural slab cache power-of-two boundaries.
Sorting Allocations by Allocator Function
To sort the slab report by the kernel functions performing the allocations (rather than the initiating call sites), use the --alloc flag:
sudo perf kmem --slab stat --alloc
You can also filter results by minimum allocation threshold or specific sorting criteria:
# Sort by total memory allocated descending
sudo perf kmem --slab stat --caller --sort total
# Sort by allocation hit count
sudo perf kmem --slab stat --caller --sort hit
# Filter for call sites allocating more than 1MB
sudo perf kmem --slab stat --caller --min-bytes 1048576
Inspecting Physical Page Allocations with –page Mode
When troubleshooting page frame exhaustion or virtual memory fragmentation, inspect Buddy allocator page allocations:
sudo perf kmem --page stat --caller
This report categorises memory by page order (where Order 0 = 4KB, Order 1 = 8KB, Order 2 = 16KB, up to Order 10 = 4MB):
------------------------------------------------------------------------------------------------------
Call site | Total alloc (bytes) | Order 0 | Order 1 | Order 2 | Order 3 | Order >= 4
------------------------------------------------------------------------------------------------------
__page_cache_alloc+0x54 | 209715200 | 51200 | 0 | 0 | 0 | 0
alloc_pages_current+0x6 | 16777216 | 0 | 0 | 1024 | 0 | 0
If high-order page allocations (Order 3 and above) show zero allocations or frequent fallbacks under memory stress, the kernel is suffering from external physical memory fragmentation, which can cause jumbo network frame allocation failures or device driver DMA allocation timeouts.
Live Interactive Tracing with Live Call Graphs
To inspect full call graph backtraces that lead to high-volume kernel allocations, record with stack unwinding enabled:
sudo perf kmem record --slab -g -- sleep 5
Then inspect the call graph hierarchy in standard reporting:
sudo perf report --stdio -n
This displays the complete call chain from user-space syscall down through the VFS and network socket layers into the exact slab cache allocator, pinpointing the application operations driving kernel memory consumption.