How to Inspect and Profile Linux Kernel Memory Allocations Using slabtop and vmstat

When investigating high RAM consumption on Linux production servers, system administrators frequently execute traditional monitoring tools such as free -m, top, or htop. However, in high-throughput database servers, virtualisation hosts, or microservice environments processing millions of concurrent network connections, these tools often present a misleading picture: total physical RAM appears exhausted, yet summing the resident set sizes (RSS) of all running user-space processes accounts for only a fraction of the consumed memory. The unaccounted memory is almost always residing within kernel space—specifically allocated across kernel slab caches. The Linux kernel uses the SLAB and SLUB memory allocators to manage pre-allocated pools of kernel objects, such as inode caches, directory dentries, network socket buffers, and task structs. Mastering the native slabtop and vmstat command-line utilities allows administrators to audit kernel memory consumption and diagnose low-level memory leaks.

Understanding Kernel Slab Allocators and Object Caching

In operating system kernel design, repeatedly allocating and freeing small kernel data structures (such as file inodes or TCP socket descriptors) via standard page-level allocators causes severe memory fragmentation and processor context-switching latency. To resolve this, the Linux kernel employs object-based slab allocation.

The slab allocator reserves continuous physical memory pages and partitions them into fixed-size slots called “slabs.” Each slab contains multiple pre-initialised object caches:

• dentry: Directory cache entries holding mappings between directory paths and inode numbers. Excessive dentry usage frequently occurs when file scanning daemons or backup tools traverse millions of files.
• inode_cache: In-memory representations of active filesystem inodes.
• kmalloc-*: General-purpose kernel memory buffers sized in powers of two (e.g. kmalloc-512, kmalloc-1024) utilised by hardware drivers and kernel modules.
• buffer_head: Descriptors holding block I/O mappings between disk storage and memory pages.

Slab memory is categorized into SReclaimable (memory that the kernel can immediately purge and reclaim under high memory pressure) and SUnreclaim (memory permanently locked by active kernel drivers or network connections). Tracking the ratio between reclaimable and unreclaimable slab memory is critical for maintaining server stability.

Evaluating System-Wide Memory Boundaries Using vmstat

Before deep-diving into individual slab objects, use vmstat to obtain an instantaneous high-level overview of virtual memory allocation and page swapping activity.

1. Open your Linux terminal and execute vmstat with memory formatting:

   vmstat -s -S M

2. Inspect the output lines detailing kernel cache allocations:
   • total memory: Total physical RAM installed on the machine.
   • buffer memory: Memory consumed by raw disk block I/O buffers.
   • swap cache: Memory tracking pages written to disk swap.
3. To review live slab allocation summary statistics directly from /proc/meminfo via vmstat, run:

   vmstat -m

   This command outputs an exhaustive tabular breakdown of every registered slab cache, listing active object counts, total available objects, object byte sizes, and active memory page allocations.

Real-Time Interactive Slab Inspection Using slabtop

The slabtop utility (provided by the procps-ng package) provides an interactive, top-like terminal dashboard that displays real-time kernel slab cache activity ordered by memory consumption.

1. Launch slabtop with administrative privileges:

   sudo slabtop

2. The header displays aggregate kernel slab telemetry:
   • Active / Total Objects: Displays the proportion of cache objects currently referenced by the kernel versus idle objects held in reserve.
   • Active / Total Slabs: Shows the physical slab page collections allocated by the memory subsystem.
   • Active / Total Size: Indicates total physical RAM consumed by the kernel slab allocator.
3. Below the header, individual kernel slab caches are displayed in descending order. Column headers include:
   • OBJS: Total number of objects currently residing within the cache.
   • ACTIVE: Number of objects currently in active use.
   • USE: Percentage of allocated object slots actively utilized (ACTIVE / OBJS * 100).
   • OBJ SIZE: The byte size of each discrete object.
   • CACHE SIZE: Total physical RAM consumed by this specific slab cache.
   • NAME: The internal kernel identifier for the object cache (e.g. dentry, ext4_inode_cache, skbuff_head_cache).

Interactive Sorting and Filtering in slabtop

While slabtop is actively running, you can alter the sort criteria dynamically using keyboard shortcuts:

• Press c: Sort by total Cache Size (identifies which kernel subsystem is consuming the most physical RAM).
• Press o: Sort by total Object count.
• Press a: Sort by number of Active objects.
• Press u: Sort by utilization percentage (identifies poorly packed or fragmented slabs).
• Press q: Exit slabtop and return to the shell.

Running in Batch Mode for Scripted Telemetry

To capture a single instantaneous snapshot of top slab consumers for automated monitoring scripts or incident reports, invoke batch mode (-o):

sudo slabtop -o | head -n 25

Diagnosing and Clearing Runaway Dentry and Inode Caches

If slabtop indicates that dentry or inode_cache consumes gigabytes of system memory, your machine is caching millions of filesystem lookup paths. While the kernel automatically reclaims these caches when user applications demand RAM, you can manually trigger kernel cache flushing during troubleshooting.

1. Force all pending dirty memory buffers to write to disk:

   sync

2. Instruct the kernel to drop clean dentry and inode caches from memory:

   echo 2 | sudo tee /proc/sys/vm/drop_caches

3. Immediately re-run sudo slabtop -o | head -n 15. Observe the dramatic reduction in dentry object counts and reclaimed RAM.

Mastering slabtop and vmstat gives system administrators complete transparency into kernel memory dynamics, preventing kernel-level memory exhaustion across high-demand Linux production environments.