In modern operating systems, memory allocation operates through demand paging. When a user-space application allocates memory using malloc() or mmap(), the Linux kernel does not immediately allocate physical RAM pages; it simply reserves virtual memory address ranges. Physical memory is mapped only when the CPU attempts to read or write to an unmapped virtual address, triggering a hardware Page Fault exception. While page faults are normal kernel operations, excessive major faults or memory compaction delays can introduce severe transaction latency. Tracing page fault latency using eBPF exposes exact paging bottlenecks in real time.
Minor vs Major Page Faults
The Linux kernel categorizes page fault events into two distinct performance tiers:
- Minor Page Faults: Occurs when the requested page already resides in physical memory—such as shared memory libraries or pages held in the kernel page cache—and merely requires updating the process page table entries (PTE). Minor faults complete rapidly in nanoseconds or low microseconds.
- Major Page Faults: Occurs when the requested data is not present in RAM and must be read from secondary storage (e.g., reading an executable binary, demand-paging an unmapped file segment, or reading from swap space). Major page faults block application threads while waiting for disk I/O, introducing latency spikes ranging from milliseconds to hundreds of milliseconds.
Prerequisites and System Setup
- Linux host running kernel 5.4 or higher with eBPF support and kernel headers installed.
- Root or
CAP_BPFprivileges. - The
bpftraceandbpfcc-toolspackages installed.
# Ubuntu / Debian installation
sudo apt update && sudo apt install bpftrace bpfcc-tools linux-headers-$(uname -r)
# RHEL / Rocky Linux installation
sudo dnf install bpftrace bcc-tools kernel-devel-$(uname -r)
Tracing Page Fault Events in Real Time with bpftrace
To inspect which processes are generating high-frequency page faults and differentiate between read and write faults, attach probes to the kernel’s fault handler tracepoints:
sudo bpftrace -e '
tracepoint:exceptions:page_fault_user {
@user_faults[comm] = count();
}
tracepoint:exceptions:page_fault_kernel {
@kernel_faults[comm] = count();
}
interval:s:5 {
time("%H:%M:%S
");
print(@user_faults);
print(@kernel_faults);
clear(@user_faults);
clear(@kernel_faults);
}'
Every 5 seconds, this script prints a summary breakdown of user-space and kernel-space page faults per process, highlighting memory-intensive workloads.
Measuring Major Page Fault Latency Distributions
Major page faults require disk I/O and represent a serious risk to latency-sensitive databases. The BCC collection includes the memleak and syscount tools, but to measure the exact latency distribution of major faults, trace the kernel’s do_swap_page() and filemap_fault() routines:
sudo bpftrace -e '
#include <linux/mm.h>
kprobe:handle_mm_fault {
@start[tid] = nsecs;
@vma[tid] = arg1;
}
kretprobe:handle_mm_fault {
$start = @start[tid];
if ($start) {
$lat_us = (nsecs - $start) / 1000;
$flags = retval;
// Check if VM_FAULT_MAJOR (0x4) occurred
if ($flags & 0x4) {
printf("%s [MAJOR FAULT] Comm: %-16s | PID: %-6d | Latency: %d us
",
time("%H:%M:%S"), comm, pid, $lat_us);
@major_fault_latency_us = hist($lat_us);
}
delete(@start[tid]);
delete(@vma[tid]);
}
}
END {
clear(@start);
clear(@vma);
}'
When major page faults occur, the console prints real-time alerts showing the latency duration. Upon terminating the script, a logarithmic histogram displays the full latency distribution.
Inspecting Page Fault Counts with System Metrics
To complement real-time eBPF traces with cumulative process counters, query pidstat:
pidstat -r 1
Review the output columns:
minflt/s:Minor page faults per second.majflt/s:Major page faults per second. A non-zero value indicates active disk swapping or continuous binary paging.
Remediation and Memory Optimization Strategies
- Lock Address Spaces with mlockall(): For low-latency applications (e.g., financial trading engines or real-time media daemons), invoke
mlockall(MCL_CURRENT | MCL_FUTURE)to pre-fault and pin all virtual pages into physical RAM, preventing page faults entirely. - Configure Transparent Huge Pages (THP): Using 2MB or 1GB huge pages reduces page table entries and TLB misses, lowering page fault frequency by up to 500x for large database heaps.
- Mitigate Swap Pressure: If major faults stem from swap reading, tune
vm.swappinessin/etc/sysctl.d/99-swap.confto prioritize keeping application heap memory resident in RAM.