How to Profile Memory Access Latency and NUMA Hit Rates in Linux Using perf mem

Modern high-performance applications—such as distributed databases, memory caches, machine learning engines, and high-frequency trading platforms—are heavily constrained by memory subsystem latency. In multi-socket Non-Uniform Memory Access (NUMA) systems and modern multi-core processors, accessing data from local L1, L2, or L3 CPU caches requires only a few nanoseconds, whereas fetching data across NUMA interconnect fabrics from a remote memory node can take hundreds of CPU cycles.

To diagnose memory performance bottlenecks, identify cache thrashing, and quantify remote NUMA penalties, the Linux kernel provides the perf mem subsystem. Built directly into the Linux perf profiling framework, perf mem uses hardware event sampling (such as Intel PEBS or AMD IBS) to capture memory load and store instructions alongside their physical memory sources, access latencies, and cache hit states.

Understanding perf mem Hardware Sampling Architecture

Standard profilers track CPU cycle counts per function, but cannot pinpoint which data structures cause pipeline stalls. The perf mem tool leverages processor-level sampling registers:

• Load and Store Sampling: Samples memory instructions at precise intervals, capturing instruction pointers, virtual data addresses, and access latency in CPU cycles.
• Data Source Decoding: Identifies where each sampled byte was serviced—L1 Data Cache, L2 Cache, L3/Last Level Cache (LLC), Local DRAM, or Remote DRAM across inter-socket interconnects (Intel UPI / AMD Infinity Fabric).
• TLB Miss Detection: Identifies whether the memory access triggered Translation Lookaside Buffer (TLB) hits, misses, or expensive hardware page-table walks.

Verifying Hardware and Kernel Support for Memory Profiling

Hardware memory sampling requires hardware performance monitoring support (Intel PEBS or AMD IBS) enabled in the host UEFI/BIOS. Verify that perf is installed on your Linux system:

perf --version

Check if your current kernel permissions permit unprivileged hardware profiling, or run commands with administrative privileges. Inspect the current perf_event_paranoid setting:

cat /proc/sys/kernel/perf_event_paranoid

For system-wide memory sampling across all processes, set the paranoid level to -1 or 1:

sudo sysctl -w kernel.perf_event_paranoid=1

Recording Memory Access Samples with perf mem record

To record memory access events for a target application or workload, use perf mem record. Specify whether you want to profile memory reads (load), memory writes (store), or both:

# Sample memory read operations for a specific command
sudo perf mem record -t load ./memory_heavy_workload

# Sample memory loads system-wide across all CPU cores for 10 seconds
sudo perf mem record -a -t load sleep 10

To capture both memory loads and stores simultaneously:

sudo perf mem record -a -t load,store -- sleep 5

During the sampling window, perf records samples directly into a local binary trace file named perf.data.

Analysing Results with perf mem report

Once data collection is complete, generate an interactive report using perf mem report:

sudo perf mem report --stdio

The standard report groups memory operations by data source and overhead percentage:

# Overhead       Samples  Memory access  Symbol                    Shared Object
# ........  ............  .............  ........................  ....................
    48.20%         12450  L1 hit         [.] hash_lookup           libdatabase.so
    24.15%          6230  L2 hit         [.] matrix_multiply       libmath.so
    14.30%          3690  L3 hit         [.] process_event_queue   app_worker
     8.40%          2170  Local RAM hit  [.] scan_memory_buffer    app_worker
     4.95%          1280  Remote RAM hit [.] scan_memory_buffer    app_worker

To inspect access latency statistics and identify operations that took longest to complete, append the --sort flag:

sudo perf mem report --sort=mem,sym,dso --stdio

Filtering Memory Access by Data Source

To pinpoint high-latency remote NUMA penalties, filter the report to focus exclusively on DRAM hits:

# Focus on Local DRAM accesses
sudo perf mem report --stdio | grep -E "Local RAM"

# Focus on Remote DRAM accesses (NUMA cross-node traffic)
sudo perf mem report --stdio | grep -E "Remote RAM"

If an application exhibits a high ratio of Remote RAM hit relative to Local RAM hit, it indicates poor NUMA thread affinity. Resolving this typically requires binding the process to local memory nodes using numactl:

numactl --cpunodebind=0 --membind=0 ./memory_heavy_workload

Inspecting Access Latency Histograms

To view the latency distribution of memory load instructions (measured in clock cycles), review the memory latency summary:

Data Source	Typical Access Latency	Performance Impact
L1 hit	~4 to 5 clock cycles	Optimal throughput; code structures fit in CPU cache lines
L2 hit	~12 to 14 clock cycles	Fast execution; minimal CPU pipeline wait
L3 hit (LLC)	~35 to 50 clock cycles	Acceptable for large datasets shared across core complexes
Local RAM hit	~150 to 200 clock cycles	Standard DRAM penalty; consider prefetching optimizations
Remote RAM hit	~250 to 400+ clock cycles	Severe NUMA interconnect penalty; bind threads to local NUMA domains

By regularly auditing application memory patterns with perf mem, systems engineers can identify cache misses, eliminate costly inter-socket memory transfers, and maximize throughput on high-core-count enterprise servers.