How to Trace Out-of-Memory Killer Events and Process Victims Using oomkill in Linux

When physical memory and swap allocations are completely exhausted on a Linux system, the kernel invokes the Out-of-Memory (OOM) Killer to terminate processes and recover system stability. While traditional approaches rely on reviewing historical kernel ring buffer logs using dmesg or journalctl after an incident occurs, eBPF-powered tracing tools such as oomkill capture termination events with negligible overhead at the exact microsecond the kernel triggers an OOM invocation.

How the Linux Kernel OOM Killer Operates

Understanding kernel memory scoring helps administrators diagnose why specific workloads are selected as termination victims:

• OOM Badness Score: The kernel calculates a score for each running process based on physical RAM consumption (RSS) and swap usage, scaled against system limits.
• OOM Adjustment (oom_score_adj): System services can adjust their survival priority by writing values between -1000 (immune from OOM kill) and +1000 (first targeted victim).
• Victim Selection: When an allocation request in alloc_pages() fails and reclaim routines yield zero pages, the kernel executes out_of_memory() and sends a SIGKILL to the process with the highest score.

Installing the BCC Tracing Toolkit

The oomkill utility is packaged as part of the BPF Compiler Collection (BCC). Install it on Debian, Ubuntu, or enterprise Linux distributions:

# Debian / Ubuntu
sudo apt update
sudo apt install -y bpfcc-tools linux-headers-$(uname -r)

# RHEL / Rocky Linux / AlmaLinux
sudo dnf install -y bcc-tools kernel-devel-$(uname -r)

Once installed, BCC tools are available in /usr/sbin or /usr/share/bcc/tools.

Tracing Live OOM Events with oomkill

To monitor system memory pressure and record OOM invocations in real time, run oomkill with administrative privileges:

sudo oomkill-bpfcc

The utility attaches an eBPF kprobe to the kernel function oom_killer_disable and tracepoints on mark_victim. When memory exhaustion triggers an OOM event, oomkill outputs structured telemetry:

Tracing OOM kills... Ctrl-C to stop.
TIME     TRIGGERED_BY  PAGE_REQ  VICTIM_PID  VICTIM_COMM   PAGES
10:14:02 node          1         14820       java          1258900
10:18:45 redis-server  1         16044       python3       842100

The output columns identify:

• TIME: Wall-clock timestamp of the memory allocation failure.
• TRIGGERED_BY: The process attempting to allocate memory that caused exhaustion.
• VICTIM_PID & VICTIM_COMM: The process ID and command name selected by the kernel for termination.
• PAGES: Total 4KB memory pages reclaimed following the termination.

Inspecting and Adjusting OOM Protection for Critical Daemons

To shield critical production services such as database engines or hypervisor agents from automated OOM termination:

1. Identify the target service process ID using pgrep:

   pgrep -x mongod

2. Inspect current badness scores in /proc:

   cat /proc/$(pgrep -x mongod)/oom_score
   cat /proc/$(pgrep -x mongod)/oom_score_adj

3. Configure persistent immunity in the service systemd unit file (/etc/systemd/system/mongod.service.d/override.conf):

   [Service]
   OOMScoreAdjust=-1000

4. Reload the systemd daemon and restart the service:

   sudo systemctl daemon-reload
   sudo systemctl restart mongod