How to Trace Pressure Stall Information (PSI) Events in Linux Using eBPF

Monitoring system health on busy production Linux hosts traditionally relies on high-level load averages reported by uptime or CPU utilization percentages from top. However, traditional load averages conflate runnable CPU tasks with uninterruptible disk sleep, obscuring whether high load stems from computational demand or stalled storage queues. The Linux kernel incorporates Pressure Stall Information (PSI)—a granular metric that measures the exact percentage of time tasks are starved waiting for CPU, memory, or block I/O. Tracing PSI threshold breaches using eBPF enables proactive detection of infrastructure degradation before applications crash.

Understanding Linux Pressure Stall Information (PSI)

PSI monitors three critical hardware subsystems: CPU, Memory, and I/O. For each resource, the kernel categorizes stall pressure into two distinct dimensions:

• Some: The percentage of wall-clock time that at least some runnable tasks were stalled waiting for the resource, even if other CPU cores or processes continued executing.
• Full: The percentage of wall-clock time that all active non-idle tasks were blocked waiting for memory or I/O, representing complete system-wide stagnation where no productive work could occur. (Note: CPU pressure supports only some, as a CPU cannot be stalled on itself).

PSI metrics are exposed via virtual files under /proc/pressure/{cpu,memory,io}, reporting 10-second, 60-second, and 300-second moving averages.

Prerequisites and Kernel Requirements

• Linux kernel version 4.20 or newer compiled with CONFIG_PSI=y (standard in Ubuntu 20.04+, RHEL 8+, Debian 11+).
• Root privileges to read virtual endpoints and compile eBPF probes.
• bpftrace and bpfcc-tools installed.

Inspecting Baseline PSI Metrics from the Command Line

Before launching real-time eBPF traces, inspect the active pressure averages across your server:

# Check memory stall pressure
cat /proc/pressure/memory

# Sample output:
# some avg10=0.04 avg60=0.01 avg300=0.00 total=482012
# full avg10=0.00 avg60=0.00 avg300=0.00 total=12400

If some avg10 exceeds 10.00, roughly 10% of execution time is wasted waiting for memory pages to swap or reclaim. If full avg10 exceeds 5.00, the system is entering severe thrashing.

Tracing PSI Stall State Changes with bpftrace

Rather than polling /proc/pressure in a periodic loop, you can trace the internal kernel functions responsible for updating stall states—specifically psi_task_change() and psi_group_change():

sudo bpftrace -e '
#include <linux/sched.h>

kprobe:psi_task_change {
    $clear = arg1;
    $set = arg2;
    // Log tasks entering uninterruptible I/O or memory stall states
    if ($set & 0x8) { // TSK_MEMSTALL flag
        printf("%s [MEM_STALL] Task: %-16s | PID: %-6d entered memory stall
",
            time("%H:%M:%S"), comm, pid);
    }
    if ($set & 0x1) { // TSK_IOWAIT flag
        // Trace high-frequency I/O wait transitions
    }
}'

Whenever an application thread enters the kernel’s memory stall tracking state (e.g., blocked inside page allocation or direct reclaim), bpftrace prints an instantaneous alert with the process name and process ID.

Triggering Automated Event Triggers via PSI File Descriptors

The Linux kernel allows user-space monitoring daemons to register epoll triggers directly against PSI virtual files using threshold triggers. You can specify a custom latency window and stall threshold:

# Syntax: <some|full> <threshold_us> <window_us>
# Example: Trigger if memory pressure exceeds 150ms of stall within a 1-second window
echo "some 150000 1000000" > /proc/pressure/memory

When this threshold is crossed, the file descriptor becomes readable, allowing system monitoring daemons to shed load, throttle background batch workers, or trigger container autoscaling before out-of-memory (OOM) kills occur.

Diagnostic Interpretations and Remediation Strategies

• Elevated CPU Some with Low Memory/IO: Indicates pure CPU core saturation. Inspect top CPU-consuming threads via top -H or scale up virtual CPU allocations.
• Elevated Memory Some accompanied by Memory Full: Confirms active thrashing. Direct reclaim and swap operations are choking the memory subsystem. Increase swap performance (zram/NVMe) or expand physical RAM.
• Elevated I/O Some: Points to storage controller queue bottlenecks or synchronous writeback stalls. Trace underlying disk latency using biosnoop or tune dirty memory writeback ratios.