How to Profile Page Faults and Virtual Memory Allocation in Linux Using perf

In Linux memory management, memory allocation performed through user space invocations such as malloc() or mmap() is fundamentally optimistic. The Linux kernel assigns virtual memory addresses to the requesting process immediately, but physical RAM frames are not committed until the virtual address space is actually accessed by a CPU instruction. When an instruction attempts to read or write to an unmapped virtual address, the CPU Memory Management Unit (MMU) halts execution and triggers a hardware page fault interrupt. Profiling these page fault events using the Linux perf subsystem allows systems engineers, database administrators, and software developers to pinpoint memory allocation latency, detect heavy swap paging bottlenecks, and identify inefficient page zeroing cycles across high-throughput workloads.

Understanding Linux Page Fault Mechanics and perf Event Classes

When the CPU triggers a page fault exception (Interrupt 14 on x86 architectures), the Linux kernel evaluates the faulted virtual address against the process memory descriptor (struct mm_struct) and its associated virtual memory areas (vm_area_struct). Depending on how the memory region is configured and whether physical storage backing is required, page faults fall into two distinct operational categories:

  • Minor Page Faults: The requested memory page is present in system memory (for example, in the kernel page cache, as an anonymous page allocated via demand paging, or shared across processes via copy-on-write), but the hardware page table entry (PTE) has not yet been populated for the calling process. The kernel satisfies the fault entirely within CPU RAM without touching block storage, creating the mapping and updating the MMU.
  • Major Page Faults: The requested memory page resides on a persistent block storage medium—such as an executable binary on an SSD, a memory-mapped database file, or a page that has been evicted to swap storage. The kernel must suspend process execution, issue asynchronous or synchronous block I/O requests to retrieve the data from disk into a page frame, and update the page tables. Major faults cause significant latency spikes compared to minor faults.

The Linux perf profiling architecture monitors page faults through two distinct mechanisms: the architectural software performance counter page-faults (along with its sub-counters minor-faults and major-faults), and the kernel memory tracepoint subsystem located under exceptions:page_fault_user and exceptions:page_fault_kernel.

Auditing System-Wide Page Fault Rates with perf stat

Before capturing detailed instruction-level profiles, evaluate the global frequency and distribution of minor and major page faults across a target workload using perf stat. Running this command isolates page fault metrics without incurring the profiling overhead associated with call graph recording:

# Measure page fault counters across a specific process workload
perf stat -e page-faults,minor-faults,major-faults ./memory_intensive_workload

# Profile system-wide page fault counters for 10 seconds
sudo perf stat -a -e page-faults,minor-faults,major-faults -- sleep 10

When analysing the output, observe the proportion between minor and major faults. High minor fault rates typically correlate with rapid heap allocation, thread allocation pools, or large memory initialization where pages are populated for the first time. Conversely, any sustained major page fault count suggests active swap thrashing or intense random reads against memory-mapped files.

Recording Page Fault Call Chains with perf record

To identify the exact user space functions, runtime allocators, or kernel routines triggering page faults, record execution profiles with full stack traces. Because page fault events occur at discrete execution points rather than periodic clock cycles, bind perf record directly to the page-faults software event:

# Record page faults with DWARF call graph unwinding for an active process
sudo perf record -e page-faults -g --call-graph dwarf -p 48210 -- sleep 15

# Record page faults for a new command invocation with frame pointer unwinding
perf record -e page-faults -g -- ./allocator_benchmark

For systems handling millions of memory accesses per second, recording every single minor fault can generate massive trace files and induce profiling overhead. You can isolate only major page faults when diagnosing disk-backed or swap latency bottlenecks:

# Record only major page faults across all running CPUs
sudo perf record -a -e major-faults -g -- sleep 30

Filtering User Space and Kernel Space Page Fault Tracepoints

For fine-grained diagnostics, the Linux kernel provides specialized tracepoints within the exception handling subsystem. These tracepoints expose whether memory faults originate during user space execution or within privileged kernel routines:

# Inspect available exception tracepoints
perf list "exceptions:*"

# Record user space page fault tracepoints
sudo perf record -e exceptions:page_fault_user -ag -- sleep 10

# Record kernel space page fault exceptions
sudo perf record -e exceptions:page_fault_kernel -ag -- sleep 10

User space exceptions highlight memory allocation bottlenecks inside application binaries, C standard libraries (such as glibc memory arenas), and managed language virtual machines (such as Java or Go runtimes). Kernel space exceptions typically occur during system call buffer copying (for example, copy_from_user or copy_to_user operations), network packet handling, or direct I/O transfers.

Analysing Trace Data and Inspecting Bottlenecks with perf report

Once the trace data is recorded to the default perf.data storage file, inspect the aggregated call stacks using perf report:

# Launch the interactive terminal UI for trace inspection
sudo perf report -g graph,0.5,caller

# Generate a non-interactive hierarchical summary
sudo perf report --stdio -n --show-total-period --percent-limit 1.0

Within the report interface, expand the call trees to pinpoint the exact code paths responsible for virtual memory allocation. Common architectural patterns include:

  • handle_mm_fault -> do_anonymous_page: Indicates demand paging of newly allocated zero-fill memory. If this function dominates user space performance, the application is frequently requesting unallocated heap space rather than reusing existing memory buffers.
  • handle_mm_fault -> do_wp_page: Represents Copy-on-Write (COW) page duplication. This commonly occurs immediately after a fork() system call when child processes attempt to write to inherited memory pages.
  • handle_mm_fault -> filemap_fault: Points to memory-mapped file access (mmap). When accompanied by major page faults, this indicates physical read operations fetching file contents from disk into the page cache.

Remediating Virtual Memory Allocation Latency

When profiling reveals that page fault overhead degrades workload throughput, apply targeted kernel and architectural optimisations:

  • Transparent Huge Pages (THP) and HugeTLB: Traditional 4 KB pages require 512 page fault resolutions to map 2 MB of memory. Leveraging Transparent Huge Pages (madvise mode) or statically pre-allocated HugeTLB pools reduces page fault counts by up to 99.8%, drastically lowering kernel exception overhead.
  • Memory Pre-Faulting (MAP_POPULATE): When allocating memory via mmap(), passing the MAP_POPULATE flag instructs the kernel to pre-fault all virtual pages during the initial system call, preventing unpredictable page fault latency during runtime request processing.
  • Custom Allocator Tuning: Modern allocators like jemalloc or mimalloc maintain thread-local caches and manage internal arenas to reuse mapped virtual memory rather than returning pages back to the kernel via brk or mmap.

Get the best tech tips delivered straight to your inbox.

Join thousands of readers mastering Apple, Google, Microsoft, and Linux.