How to Deploy systemd-oomd for Granular Out-Of-Memory (OOM) Process Termination

In high-density Linux environments—such as Kubernetes nodes, rendering farms, or massive database servers—memory exhaustion is a catastrophic event. Historically, when a Linux system runs out of physical RAM and swap space, the kernel invokes its legacy Out-Of-Memory (OOM) Killer. The kernel’s OOM Killer is a blunt instrument; it halts the entire operating system, calculates a heuristic score (oom_score) based on memory usage, and instantly sends a SIGKILL to the process it deems most responsible. This often results in critical system daemons (like SSH or logging agents) being terminated unexpectedly, leaving the server in an unrecoverable, deadlocked state. To solve this, modern Linux distributions (such as Fedora and Ubuntu) have adopted systemd-oomd, a highly intelligent, user-space daemon designed to take corrective action before the kernel panics.

The Architecture of systemd-oomd

Unlike the kernel OOM killer, which only reacts when the system is completely devoid of memory, systemd-oomd acts preventatively. It leverages the Linux kernel’s PSI (Pressure Stall Information) interface. PSI provides precise, real-time metrics on how long CPU, memory, and I/O tasks are waiting (stalling) for resources.

Instead of killing arbitrary, individual processes based on raw memory size, systemd-oomd monitors the PSI memory pressure across cgroups (control groups). If a specific cgroup (for example, a systemd service containing a group of worker threads, or a user slice containing a desktop session) exceeds a defined memory pressure threshold for a sustained duration, systemd-oomd will terminate the entire cgroup gracefully.

This cgroup-centric approach ensures that a runaway web application is killed entirely, rather than just one of its child processes, which would simply respawn and continue thrashing the disk.

Enabling and Verifying systemd-oomd

On modern distributions like Ubuntu 22.04 LTS and Fedora 34+, systemd-oomd is installed by default but may require explicit enablement depending on the server profile.

To verify if the daemon is active, execute:

systemctl status systemd-oomd

If it is not running, enable and start it:

sudo systemctl enable --now systemd-oomd

To view the current memory pressure metrics and the cgroups actively being monitored by the daemon, utilize the oomctl utility:

oomctl

This command outputs the current swap usage and the PSI memory pressure. It also lists the specific cgroups (e.g., /user.slice/user-1000.slice) that are eligible for termination if memory exhaustion occurs.

Configuring OOM Policies for Systemd Services

The true power of systemd-oomd is the ability to configure granular termination policies on a per-service basis. If you have a critical database service (e.g., PostgreSQL) that should never be targeted, and a batch-processing worker service that can be safely killed and restarted, you can configure this directly in their systemd unit files.

To configure a service to be actively managed by systemd-oomd, you must modify its unit file (e.g., /etc/systemd/system/batch-worker.service) and add the ManagedOOMMemoryPressure directive.

[Unit]
Description=Background Batch Processing Worker

[Service]
ExecStart=/usr/bin/python3 /opt/worker.py
# Allow systemd-oomd to kill this cgroup if it causes excessive memory pressure
ManagedOOMMemoryPressure=kill
# Specify the threshold: kill if memory pressure exceeds 50% for 20 seconds
ManagedOOMMemoryPressureLimit=50%

Conversely, for highly critical system services where termination would cause a total server outage, you can explicitly protect them. While systemd-oomd generally avoids system slices, you can enforce protection utilizing the standard kernel OOM adjustment setting in the unit file:

[Service]
ExecStart=/usr/bin/postgres
# Strongly discourage any OOM killer (kernel or systemd) from targeting this service
OOMScoreAdjust=-1000

After modifying any unit files, reload the systemd daemon to apply the new cgroup configurations:

sudo systemctl daemon-reload
sudo systemctl restart batch-worker.service

By migrating from the unpredictable kernel OOM killer to the deterministic, cgroup-aware systemd-oomd, Linux administrators can ensure that localized memory leaks never result in complete system lockups, maintaining absolute stability for mission-critical infrastructure.