How to Trace Kernel Memory Leaks and Allocation Call Paths Using memleak in Linux

Memory leaks in Linux production environments can cause gradual system degradation, culminating in Out-Of-Memory (OOM) killer invocations that terminate vital server workloads. While utilities such as free and top show total memory exhaustion, they lack the granularity to attribute memory growth to specific allocation functions or code paths. The memleak utility, part of the BPF Compiler Collection (BCC), attaches eBPF probes to memory allocation and free events, tracking outstanding allocations to pinpoint leaks in real time.

How memleak Identifies Leaked Memory

The memleak tool instruments memory allocator functions—such as user-space malloc(), calloc(), posix_memalign(), and kernel-space kmalloc(), kmem_cache_alloc(). When an allocation occurs, memleak records the memory address, allocation size, and full call stack. When a corresponding free() or kfree() is called, the record is cleared. By filtering for allocations that persist beyond an age threshold without being freed, memleak isolates leaked memory blocks with their originating stack traces.

Installing the BCC Tracing Toolkit

Ensure that BCC tools and matching kernel development headers are installed:

# On Debian or Ubuntu:
sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

# On RHEL, AlmaLinux, Rocky Linux, or Fedora:
sudo dnf install bcc-tools kernel-devel-$(uname -r)

On Debian/Ubuntu distributions, run the command as memleak-bpfcc; on RHEL-based distributions, it is located at /usr/share/bcc/tools/memleak.

Tracing Leaks in a Specific User-Space Process

To detect memory leaks within a designated user-space service (such as an Nginx or PostgreSQL worker process):

# Monitor PID 3450, reporting allocations older than 5000 milliseconds:
sudo memleak-bpfcc -p 3450 --older 5000

The --older 5000 flag instructs memleak to ignore short-lived transient allocations and only display allocations that have remained unreleased for more than 5 seconds.

Example terminal output:

[14:22:10] Top 2 stacks with outstanding allocations:
8192 bytes in 2 allocations from stack
  my_data_parser+0x42 [/usr/bin/custom_daemon]
  worker_thread_loop+0x18a [/usr/bin/custom_daemon]
  start_thread+0xd9 [libpthread.so.0]

32768 bytes in 8 allocations from stack
  buffer_expand+0x65 [/usr/bin/custom_daemon]
  process_incoming_packet+0x210 [/usr/bin/custom_daemon]

Tracing Kernel-Level Memory Allocations

To trace kernel memory allocations (SLAB/SLUB caches) and identify kernel module or driver memory leaks, execute memleak without specifying a user PID:

sudo memleak-bpfcc -a -c 5 --older 10000

Here, the -a flag includes allocations with call stacks, while -c 5 displays the top 5 largest allocation sites. The output highlights the exact kernel function, such as alloc_skb_with_frags or __kmalloc, responsible for the outstanding memory footprint.

Remediation Strategies Based on Trace Stacks

With precise call stacks identified, developers and systems engineers can apply targeted fixes:

• Unbalanced Buffer Allocation: If stack traces point to buffer expansion functions, inspect error handling branches where functions return early without releasing temporary buffers.
• Driver Leaks: Outstanding kernel allocations pointing to third-party kernel drivers indicate missing deallocations in hardware disconnect or interrupt handling routines.

Deploying memleak provides definitive diagnostic transparency, saving hours of guesswork and preventing production outages caused by memory exhaustion.