How to Trace CPU Last Level Cache (LLC) Miss Rates in Linux Using llcstat

Modern high-performance computing workloads—such as high-frequency financial trading, large-scale in-memory databases, and distributed graph analytics—frequently experience performance degradation caused by memory latency rather than raw CPU instruction throughput. While CPUs execute instructions in fractions of a nanosecond, fetching data from main system DRAM requires roughly 50 to 100 nanoseconds. The CPU Last Level Cache (LLC, commonly L3 cache) serves as the hardware’s final barrier before expensive DRAM bus cycles are triggered. When a program exhibits poor data locality or thrashes memory, frequent LLC misses stall processor pipelines. The llcstat tool from the BCC (BPF Compiler Collection) suite instruments hardware performance counters and in-kernel eBPF maps to measure LLC references, hits, and miss percentages per process.

How llcstat Instruments Hardware Performance Counters

Unlike software tracepoints that monitor operating system functions, cache misses are physical hardware events tracked by CPU Performance Monitoring Units (PMUs). The llcstat utility programs PMU hardware counters to sample two specific hardware events:

• LLC References: The total number of memory requests that reached the Last Level Cache.
• LLC Misses: The count of requests that failed to find requested data within the LLC, forcing the memory controller to read lines directly from system DRAM.

Whenever hardware counters trigger a sample, llcstat invokes an in-kernel eBPF program to associate the event with the executing thread, process ID (PID), command name, and CPU core. Much like tracking page cache efficiency with cachetop or tracing kernel memory pressure using slabratetop, llcstat provides essential hardware-level visibility into memory-bound applications.

Installing Prerequisites and BCC Packages

To run llcstat, your system requires kernel headers matching your active Linux kernel and the BCC tools package. On Debian and Ubuntu systems, run:

sudo apt update && sudo apt install -y bpfcc-tools linux-headers-$(uname -r)

On Red Hat Enterprise Linux, Rocky Linux, or Fedora, install the tools via dnf:

sudo dnf install -y bcc-tools kernel-devel-$(uname -r)

The binary is typically located at /usr/sbin/llcstat or /usr/sbin/llcstat-bpfcc. Because accessing hardware performance monitoring counters and attaching eBPF programs requires low-level kernel rights, running llcstat requires root or sudo execution privileges.

Running llcstat to Measure Cache Misses

To collect a system-wide trace of Last Level Cache performance, specify a sample duration in seconds. For example, to record cache behaviour across all running processes for 10 seconds, execute:

sudo llcstat 10

While tracing is active, the tool gathers hardware counter events with minimal system overhead. After 10 seconds, it prints a sorted breakdown of processes exhibiting the highest cache activity:

Running for 10 seconds...
PID      NAME             CPU     REFERENCE         MISS    HIT%
18420    redis-server     03       41208000       824160   98.00%
9812     mysqld           07      184500000      9225000   95.00%
1204     nginx            01       12040000       120400   99.00%
24102    ffmpeg           05      890100000    178020000   80.00%
312      kworker/u16:1    02         540000        10800   98.00%

The columns reveal critical memory access characteristics:

• PID / NAME: Process ID and executable command name running on the processor.
• CPU: Physical or logical CPU core where the process was scheduled during the sampling interval.
• REFERENCE: Estimated total LLC cache lookups requested by the workload.
• MISS: Estimated count of cache line misses that triggered external DRAM access.
• HIT%: Calculated cache hit efficiency (100 - (MISS / REFERENCE * 100)).

Configuring Sampling Frequency and Continuous Output

By default, llcstat samples hardware PMU counters at a default rate (100 events). If you are running high-performance benchmarks or need continuous periodic updates, pass an interval and iteration count:

sudo llcstat -c 500 5

The -c flag adjusts the sample period to one counter interrupt every 500 events, reducing interrupt overhead during intense multi-threaded workloads while refreshing output tables every 5 seconds.

Diagnosing Memory-Bound Workloads and NUMA Contention

When an application exhibits an LLC hit rate falling below 85% with high reference volume, the CPU is frequently stalling on memory bus wait cycles. Common remedies include:

• Data Locality Optimisation: Restructuring application data structures from array-of-structures (AoS) to structure-of-arrays (SoA) to maximize cache line prefetching.
• NUMA Pinning: Binding memory-intensive threads to specific CPU sockets using numactl --cpunodebind and --membind to prevent cross-socket interconnect latency.
• Thread Contention Analysis: Combining llcstat findings with offcputime blocking latency traces and network traffic distribution with tcpsubnet to distinguish between thread lock contention, network wait states, and physical cache exhaustion.

Press Ctrl + C at any time to conclude data collection and print intermediate summary statistics.