How to Trace User-Space Memory Leaks in Linux Using memleak

Memory leaks in user-space Linux applications can degrade system responsiveness, trigger thrashing in the swap subsystem, and ultimately force the kernel’s Out-Of-Memory (OOM) killer to terminate critical background daemons. Traditional memory profilers such as Valgrind work by emulating instruction execution within an instrumented virtual CPU, introducing severe 20x to 50x performance penalties that make them unusable in production environments. The memleak tool, part of the BPF Compiler Collection (BCC), solves this diagnostic hurdle by attaching lightweight eBPF probes directly to dynamic memory allocation and deallocation calls (such as malloc, calloc, realloc, posix_memalign, and free) in user space or the C runtime library (glibc). This article demonstrates how to trace active memory leaks, capture allocation call stacks, and isolate unreleased memory buffers in Linux using memleak.

Understanding eBPF Memory Allocation Tracing with memleak

Unlike kernel memory leak detectors like kmemleak which scan slab caches and page allocations, memleak monitors user-space virtual memory management. Whenever an instrumented process requests memory through the standard C library allocator or raw system calls (such as mmap and brk), memleak intercepts the entry and exit of the allocation function:

• Allocation Tracking: Records the memory address, allocation size in bytes, timestamp, and the exact user-space stack trace leading up to the request.
• Deallocation Matching: Intercepts corresponding calls to free() or munmap(), removing matching memory pointers from the internal BPF hash table.
• Outstanding Leak Detection: Periodically scans the hash table for allocations that have persisted without being freed past a specified age threshold, aggregating outstanding allocations by their unique stack trace call paths.

Because eBPF probes execute in kernel space and only extract lightweight pointer metadata, memleak imposes minimal runtime overhead, making it safe for diagnostic troubleshooting on active staging and production nodes.

Installing BCC Tools and Kernel Debug Symbols

To run memleak, ensure the BCC toolchain and kernel development packages are installed. On Debian and Ubuntu distributions, execute:

sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

On Red Hat Enterprise Linux, Rocky Linux, AlmaLinux, or Fedora systems, install the equivalent packages:

sudo dnf install bcc-tools kernel-devel-$(uname -r)

Additionally, to resolve hexadecimal instruction pointers into human-readable function names and source line numbers within stack traces, verify that debug symbols or unstripped binaries are accessible for your target application.

Tracing Memory Leaks in a Specific Running Process

To inspect a process suspected of leaking memory, identify its Process ID (PID) using pgrep or top, then execute memleak-bpfcc with the -p parameter:

sudo memleak-bpfcc -p 14820

By default, memleak prints outstanding allocations every 5 seconds, filtering out allocations that occurred within the last 500 milliseconds to avoid reporting temporary working buffers. A representative terminal output displays the following call trace:

[14:22:10] Top 2 stacks with outstanding allocations:
8192 bytes in 64 allocations from stack
  0x00007f9c2d1045a2 malloc+0x42 [/lib/x86_64-linux-gnu/libc.so.6]
  0x000055c18a2014e8 buffer_create+0x28 [/usr/local/bin/worker_daemon]
  0x000055c18a201a14 process_incoming_job+0x134 [/usr/local/bin/worker_daemon]
  0x000055c18a20220c main+0x10c [/usr/local/bin/worker_daemon]
  0x00007f9c2d0e1d90 __libc_start_main+0xf0 [/lib/x86_64-linux-gnu/libc.so.6]

The output reports the accumulated leaked byte total (8,192 bytes), the number of separate outstanding allocations (64), and the exact call path (main -> process_incoming_job -> buffer_create -> malloc), pointing directly to the leaking code routine.

Targeting Specific Allocation Age and Size Thresholds

In high-throughput services, thousands of temporary allocations are created and freed every second. To prevent active, legitimate working memory from cluttering the output, tune the age and size filters:

1. Filter by Allocation Age (-a): To ignore recent short-lived buffers and display only allocations that have remained unreleased for at least 30 seconds (indicating persistent memory retention), run:

sudo memleak-bpfcc -p 14820 -a 30

2. Filter by Minimum Allocation Size (-s): To ignore small micro-allocations and focus exclusively on large buffers exceeding 64 KB (65,536 bytes), execute:

sudo memleak-bpfcc -p 14820 -s 65536

3. Configure Sampling Interval (-i): To sample memory state every 10 seconds for 6 iterations (spanning 1 minute of profiling), specify the interval and count:

sudo memleak-bpfcc -p 14820 -i 10 6

Tracing Leaks from Process Launch and glibc Alternatives

When debugging applications that leak memory during initialisation or short-lived batch jobs, execute the target process directly under memleak using the -c command wrapper:

sudo memleak-bpfcc -c "./data_processor --input batch.dat"

If your application links dynamically against a custom memory allocator (such as Jemalloc or TCMalloc) instead of the standard glibc allocator, instruct memleak to attach probes to the specific shared library using the -O flag:

sudo memleak-bpfcc -p 14820 -O /usr/lib/x86_64-linux-gnu/libjemalloc.so.2

Tracing Raw Kernel Memory Allocations with memleak

While memleak is primarily utilized for user-space profiling, it can also attach to kernel-space allocator functions (such as kmalloc, kmem_cache_alloc, and vmalloc) without requiring recompilation with CONFIG_KMEMLEAK. To trace kernel leaks globally across all running subsystems, pass the -k flag:

sudo memleak-bpfcc -k -a 60

By leveraging memleak to trace user-space and kernel allocations, Linux system administrators and developers can rapidly pinpoint memory leaks in live workloads before they trigger system instability or Out-Of-Memory termination.