How to Trace High-Latency Page Faults and Paging Activity Using faultslower in Linux

In high-throughput databases, low-latency trading systems, and real-time backend microservices, brief micro-stalls in application responsiveness can degrade overall cluster performance. While page faults are a normal part of virtual memory management—allocating physical RAM on demand or paging data from disk—major page faults that block on synchronous disk I/O can introduce multi-millisecond latencies into critical execution threads. The faultslower utility, built upon eBPF in the BCC toolkit, identifies and logs high-latency page faults in real time.

Minor vs. Major Page Faults and Performance Impact

Understanding the distinction between page fault types is essential for diagnosing memory bottlenecks:

• Minor Page Faults: The requested memory page exists in physical memory (e.g., shared library cache or newly allocated zero-fill page) but lacks a mapped virtual memory address. These resolve quickly in CPU cycles without disk access.
• Major Page Faults: The requested page does not reside in RAM and must be read synchronously from secondary storage (such as swap space or a memory-mapped file). These block the calling thread for milliseconds.

Installing the BCC Tracing Toolkit

Ensure that BCC tools and kernel development headers are installed on your Linux system:

# On Debian or Ubuntu:
sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

# On RHEL, AlmaLinux, Rocky Linux, or Fedora:
sudo dnf install bcc-tools kernel-devel-$(uname -r)

On Debian/Ubuntu environments, execute the binary as faultslower-bpfcc; on enterprise RHEL distributions, find it at /usr/share/bcc/tools/faultslower.

Tracing High-Latency Faults Across the Entire System

To capture page faults that take longer than a defined latency threshold (e.g., 5 milliseconds), run faultslower with root privileges:

sudo faultslower-bpfcc 5

Example terminal output:

TIME(s)     COMM           PID    ADDR               M LAT(ms)
0.000000    mysqld         2410   0x7f81a4b2c000     M    8.42
0.012540    redis-server   1890   0x7fff5e2a1000     m    5.10
0.142050    postgres       3104   0x7f3320c15000     M   14.28

In this output:

• COMM & PID: The executable name and process identifier encountering the stall.
• ADDR: The virtual memory address triggering the fault.
• M (Type): M denotes a Major page fault (disk read), whereas m denotes a Minor page fault.
• LAT(ms): The duration the calling thread was suspended waiting for kernel memory resolution.

Targeting a Specific High-Priority Process

When tuning a latency-sensitive service, isolate its Process ID (PID) to eliminate background noise:

# Trace page faults slower than 1 millisecond for PID 1890:
sudo faultslower-bpfcc -p 1890 1

Eliminating Root Causes of Page Fault Stalls

Armed with precise telemetry from faultslower, engineers can apply targeted mitigations:

• Memory Locking (mlockall): For real-time applications, pin critical memory pages into physical RAM using mlock() or configuration directives (e.g., vm.max_map_count and Redis memlock limits).
• Adjusting Swappiness: Frequent major faults on active heap memory indicate excessive paging; reduce vm.swappiness (e.g., sysctl vm.swappiness=10) to prioritize application RAM over file cache.
• Transparent Huge Pages (THP): If minor faults take excessive time during high-frequency allocation loops, evaluate whether disabling THP (echo never > /sys/kernel/mm/transparent_hugepage/enabled) reduces page table allocation latency.

Using faultslower eliminates virtual memory blind spots, allowing administrators to stabilise application response times across demanding Linux workloads.