How to Profile CPU Cache Misses and Memory Latency in Linux Using perf stat

Modern microprocessors operate at clock frequencies measured in gigahertz, executing instructions in fractions of a nanosecond. However, dynamic random-access memory (DRAM) access speeds have not kept pace with CPU instruction throughput—a fundamental bottleneck known as the memory wall. While a Level 1 (L1) data cache hit resolves in approximately 4 to 5 CPU cycles, fetching a data line from main RAM requires 150 to 250 cycles. When CPU execution pipelines stall waiting for cache line fills, computational performance plummets.

Diagnosing whether an application’s poor throughput is caused by algorithmic complexity or memory access stalls requires low-level hardware performance counters. Through the hardware Performance Monitoring Unit (PMU) integrated into modern Intel, AMD, and ARM processors, the Linux kernel profiling subsystem provides deep visibility into cache efficiency. By using perf stat with targeted cache hardware events, systems engineers can quantify L1 data cache misses, Last-Level Cache (LLC) line invalidations, and memory bus saturation directly from the command line.

Understanding the CPU Cache Hierarchy and PMU Events

Modern CPUs organize cache memory into hierarchical tiers designed to balance speed, size, and electrical power:

• L1 Data Cache (L1d): Dedicated per-core cache (typically 32 KB to 48 KB) offering the lowest access latency (~1 nanosecond). Events include L1-dcache-loads, L1-dcache-load-misses, and L1-dcache-stores.
• L2 Cache: Dedicated per-core or shared cache (typically 512 KB to 1 MB) acting as a secondary buffer (~3 to 4 nanoseconds). Events include l2_rqsts.all_demand_data_rd and l2_rqsts.miss.
• Last-Level Cache (LLC / L3): Massive shared on-die cache (typically 16 MB to 96 MB+) shared across all CPU cores on a socket (~10 to 15 nanoseconds). Events include LLC-loads, LLC-load-misses, LLC-stores, and LLC-store-misses.
• Main System RAM: External memory accessed over the memory controller bus when an LLC miss occurs.

A cache miss rate (calculated as cache-misses / cache-references * 100) exceeding 10% on an intensive loop strongly suggests poor data locality, non-contiguous memory access, or false sharing across threads.

Profiling Overall Cache Misses with perf stat

To inspect generalized hardware cache references and misses across a specific command or workload, invoke perf stat using standard architectural event aliases:

# Profile generalized hardware cache metrics for a command
sudo perf stat -e cache-references,cache-misses,cycles,instructions -- ./database_bench

The command outputs hardware event counts along with calculated ratios:

 Performance counter stats for './database_bench':

     1,845,120,910      cache-references
       210,481,200      cache-misses              #   11.41% of all L1D hits
     8,412,019,450      cycles
    12,891,440,210      instructions              #    1.53  insn per cycle

       2.145019821 seconds time elapsed

Here, the cache miss rate stands at 11.41%. Notice the instructions per cycle (IPC) score of 1.53; if the cache miss rate were significantly higher (e.g. 30%), the IPC would plummet below 0.6 due to memory stalls.

Measuring L1 Data Cache and Last-Level Cache (LLC) Hierarchy

To isolate exactly where memory bottlenecks occur within the cache hierarchy, specify granular hardware cache events covering both L1 and LLC layers:

# Profile L1 and LLC load miss rates simultaneously
sudo perf stat -e \
    L1-dcache-loads,L1-dcache-load-misses,\
    LLC-loads,LLC-load-misses \
    -p 18452 -- sleep 10

The terminal presents hierarchical breakdown metrics:

 Performance counter stats for process id '18452':

     3,410,290,110      L1-dcache-loads
       185,420,109      L1-dcache-load-misses     #    5.44% of all L1-dcache hits
        42,109,820      LLC-loads
         5,120,401      LLC-load-misses           #   12.16% of all LL-cache hits

      10.001290141 seconds time elapsed

In this profile:

• 94.56% of data requests hit the L1 data cache directly.
• Of the data requests that missed L1 and reached the LLC (42.1 million requests), only 12.16% missed the LLC, meaning only 5.12 million requests required expensive main RAM access over the 10-second period.

System-Wide Cache Profiling Across CPU Sockets

On multi-socket NUMA servers hosting virtualization or database clusters, you can profile cache performance aggregated across all CPU cores or segmented by socket:

# Profile cache misses aggregated per physical CPU socket
sudo perf stat -a -A -e cache-misses,cache-references -- sleep 5

The -A flag displays statistics for each CPU core independently:

CPU0      2,145,200      cache-misses
CPU1      2,091,400      cache-misses
CPU2      8,451,900      cache-misses
CPU3      8,120,100      cache-misses

Significant disparities between cores highlight asymmetric workload distribution or NUMA memory node misalignments where a thread on socket 0 is accessing memory allocated on socket 1.

Pinpointing Cache-Intensive Code with perf record

Once perf stat confirms that high cache misses are hurting performance, switch to perf record targeting the cache-misses event to identify the exact functions and machine instructions responsible:

# Sample the hottest instruction addresses causing cache misses
sudo perf record -e cache-misses -g -p 18452 -- sleep 15

# View the interactive report sorted by cache miss percentage
perf report -i perf.data

In the interactive browser, drill down into the top functions and press a to disassemble the code. Instructions associated with memory pointers (such as mov (%rax), %rcx) showing large percentages indicate where cache-line unaligned structures or pointer chasing are occurring.

Using perf stat to profile CPU cache hierarchies empowers Linux software developers and performance engineers to optimize data layout, restructure loops for cache blocking, and eliminate expensive memory access stalls.