How to Detect and Diagnose Kernel Memory Leaks in Linux Using kmemleak

Debugging memory consumption issues on Linux typically begins in user space with tools like top, htop, or valgrind. However, when physical memory steadily decreases while user processes consume minimal RAM, the leak resides within the Linux kernel itself. Kernel memory leaks commonly stem from third-party out-of-tree kernel modules, buggy hardware device drivers, network socket buffers that fail to free sk_buff allocations, or file system filter drivers that orphan slab objects.

To detect and isolate kernel memory leaks without attaching invasive hardware debuggers, the Linux kernel includes an integrated memory tracking subsystem: kmemleak (the Kernel Memory Leak Detector). kmemleak operates similarly to a mark-and-sweep garbage collector, continuously recording memory allocations made via kmalloc(), kmem_cache_alloc(), vmalloc(), and tracking unreferenced memory pointers. By inspecting /sys/kernel/debug/kmemleak, performance engineers and system administrators can pinpoint the exact kernel functions and source code call stacks causing memory loss. This guide explains how kmemleak operates, how to verify kernel support, and how to trigger scans, filter leak reports, and clear false positives in Linux.

Architecture of the Kernel Memory Leak Detector (kmemleak)

The Linux kernel’s kmemleak subsystem tracks dynamic memory allocations across kernel slab and vmalloc memory pools:

• Allocation Tracking: Whenever a kernel subsystem allocates memory (e.g. via kmalloc()), kmemleak intercepts the call, records the pointer address, allocation size, and backtrace (call stack), and marks the object as potentially unreferenced.
• Memory Scanning: During a scan, kmemleak examines all memory regions that could hold pointers to allocated objects—including the kernel data segment, task stacks, CPU registers, and allocated memory blocks. If an allocated object’s address cannot be found anywhere in memory, it is classified as an orphan (leak).
• Min-Count Thresholds: To prevent false positives caused by transient data structures or delayed garbage collection, kmemleak requires an object to remain unreferenced across multiple scan cycles before flagging it as a confirmed leak.
• DebugFS Interface: kmemleak exposes its control interface and human-readable diagnostic reports through /sys/kernel/debug/kmemleak.

Because kmemleak maintains an internal metadata record for every active kernel allocation, it introduces minor CPU and memory overhead (typically 10–20% additional RAM usage) and should be enabled primarily during testing, staging, and driver validation.

Verifying Kernel Configuration and Enabling kmemleak

kmemleak requires specific kernel compilation flags. Check whether your active Linux kernel build includes kmemleak support:

# Check kernel config for KMEMLEAK flags
zcat /proc/config.gz 2>/dev/null || grep -E "CONFIG_DEBUG_KMEMLEAK" /boot/config-$(uname -r)

The output must confirm that kmemleak is compiled into the kernel:

CONFIG_DEBUG_KMEMLEAK=y
CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF=n

If CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF=y is set, kmemleak is built but disabled by default to save memory. You can enable it at boot by appending kmemleak=on to your kernel command line in GRUB (/etc/default/grub) and running sudo update-grub.

Next, ensure that debugfs is mounted, as kmemleak communicates through the debugfs virtual filesystem:

# Verify debugfs mount
mount | grep debugfs

# Mount debugfs manually if absent
sudo mount -t debugfs none /sys/kernel/debug

Verify that the kmemleak node exists:

ls -l /sys/kernel/debug/kmemleak

Triggering an On-Demand Kernel Memory Scan

By default, kmemleak runs an automatic background scan every 10 minutes. However, during diagnostic testing—such as before and after running a high-load network test or inserting a device driver—you can trigger an immediate manual scan.

To initiate a memory scan, write scan to the kmemleak control node:

echo scan | sudo tee /sys/kernel/debug/kmemleak

The kernel will iterate through all memory blocks, scanning memory pointers across all CPU cores. Depending on system RAM capacity, the scan typically completes within 1 to 5 seconds.

Reading and Interpreting kmemleak Diagnostic Reports

Once a scan completes, inspect the output report by reading /sys/kernel/debug/kmemleak:

sudo cat /sys/kernel/debug/kmemleak

If no memory leaks are detected, the file returns empty. If unreferenced memory blocks are identified, kmemleak outputs structured leak entries detailing the memory address, allocation size, timestamp, and the full kernel execution call trace:

unreferenced object 0xffff888114a82100 (size 256):
  comm "modprobe", pid 1482, jiffies 4294967320 (age 142.120s)
  hex dump (first 32 bytes):
    00 00 00 00 00 00 00 00 80 21 a8 14 81 88 ff ff  .........!......
    40 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  @...............
  backtrace:
    [<ffffffff812bc410>] __kmalloc+0x180/0x320
    [<ffffffffa0012080>] custom_driver_init+0x50/0x120 [custom_driver]
    [<ffffffff81002194>] do_one_initcall+0x44/0x200
    [<ffffffff811f0412>] do_init_module+0x52/0x220
    [<ffffffff811f2680>] load_module+0x1680/0x2400
    [<ffffffff811f3540>] __do_sys_finit_module+0xb0/0x110
    [<ffffffff81c0021b>] do_syscall_64+0x5b/0x90
    [<ffffffff81e00072>] entry_SYSCALL_64_after_hwframe+0x63/0xcd

Analyzing this diagnostic entry provides direct troubleshooting answers:

• Unreferenced Object & Size: Identifies the orphaned memory address (0xffff888114a82100) and its size (256 bytes).
• Process Attribution (comm and pid): Shows that process modprobe with PID 1482 triggered the allocation.
• Backtrace: Reveals that inside the kernel module custom_driver, the function custom_driver_init allocated 256 bytes via __kmalloc but failed to store or track the resulting pointer, causing a permanent leak upon module initialization.

Configuring kmemleak Runtime Parameters

kmemleak supports several operational commands passed directly into /sys/kernel/debug/kmemleak to control scanning behavior:

Adjusting the Automatic Scan Interval

To change how often kmemleak performs automatic background scans (specified in seconds):

# Set automatic scan interval to 60 seconds (default is 600)
echo "scan=60" | sudo tee /sys/kernel/debug/kmemleak

# Disable automatic scanning completely (manual scan only)
echo "scan=off" | sudo tee /sys/kernel/debug/kmemleak

Clearing the Leak Log Buffer

After reviewing reported leaks or fixing an offending driver, clear the diagnostic buffer so previous entries do not clutter future investigations:

echo clear | sudo tee /sys/kernel/debug/kmemleak

Filtering False Positives

In some situations, a driver may purposefully store pointers in non-standard ways (such as XOR-encoded pointers or custom memory pools) that kmemleak’s scanner cannot resolve. If you verify an allocation is intentional, instruct kmemleak to ignore false positives or dump raw memory contents:

# Dump current memory references for a specific address
echo "dump=0xffff888114a82100" | sudo tee /sys/kernel/debug/kmemleak

Disabling kmemleak to Reclaim System Overhead

When diagnostic investigations and driver validation are complete, stop the kmemleak subsystem to eliminate memory tracking overhead and free its metadata tables:

# Stop memory tracking
echo stack=off | sudo tee /sys/kernel/debug/kmemleak
echo off | sudo tee /sys/kernel/debug/kmemleak

Once disabled, kmemleak ceases recording allocations until the next system reboot, restoring maximum kernel throughput for production workloads.