How to Detect and Diagnose CPU Cache False Sharing in Linux Using perf c2c

In high-concurrency multi-threaded software (such as database engines, high-frequency trading platforms, and distributed network proxies), developers frequently observe multi-core performance scaling plateauing or collapsing despite low CPU utilization. A primary architectural cause of this degradation is False Sharing. Modern x86 and ARM processors do not read and write individual bytes in main memory; instead, memory is transferred in chunks called cache lines (typically 64 bytes). When two CPU cores independently modify separate variables that reside within the identical 64-byte cache line, hardware cache coherency protocols (MESI/MOESI) invalidate the entire cache line across cores, forcing expensive bus snooping and memory stalls. Linux provides perf c2c (Cache-to-Cache), a specialized profiler designed to isolate false sharing and cache-line contention.

Understanding Cache Coherency and the False Sharing Phenomenon

To diagnose memory stalls effectively, systems engineers must understand how hardware cache coherency operates across multi-core processors:

• Cache Lines: The fundamental unit of data transfer between L1/L2/L3 caches and system RAM (64 bytes).
• Modified State (M) & Invalidation: When Core 0 writes to Variable A, its local cache marks the cache line as Modified. If Core 1 simultaneously reads or writes to unrelated Variable B on that same 64-byte boundary, Core 1’s cache line is marked Invalid.
• HITM (Hit in Modified Cache): When a core requests data that is currently dirty in another core’s cache, the request results in a HITM event. In Non-Uniform Memory Access (NUMA) systems, a Remote HITM across CPU sockets takes hundreds of clock cycles, severely bottlenecking multi-threaded throughput.
• perf c2c Engine: Leverages hardware Performance Monitoring Counters (PMC) and Processor Event-Based Sampling (PEBS on Intel or IBS on AMD) to record memory loads and stores, quantifying precise HITM latency per cache line.

Prerequisites and Kernel Tracing Permissions

Executing perf c2c requires the Linux performance analysis suite (linux-tools) and appropriate kernel event sampling privileges:

# Install Linux perf tools on Debian/Ubuntu
sudo apt update && sudo apt install -y linux-tools-common linux-tools-generic linux-tools-$(uname -r)

# Configure kernel paranoid level to allow user-space profiling
sudo sysctl -w kernel.perf_event_paranoid=-1

# Ensure kernel memory addresses are visible
sudo sysctl -w kernel.kptr_restrict=0

Recording Cache-to-Cache Access Events with perf c2c record

To record memory access profiles across a running application or benchmark workload, execute perf c2c record. For example, profile a multi-threaded binary named worker_node:

# Record cache-to-cache access events across all threads of a process
sudo perf c2c record -F 60000 -- ./worker_node --threads 16

To attach to an already running production daemon by its process ID (PID):

# Attach perf c2c to an active multi-threaded process for 30 seconds
sudo perf c2c record -p 14205 -- sleep 30

The command samples load and store latency, storing the trace data in perf.data.

Analyzing Cache Line Contention with perf c2c report

Once the trace capture is complete, generate the diagnostic report using perf c2c report:

# Generate interactive terminal cache-to-cache report
sudo perf c2c report --stdio

The output is partitioned into two critical summary tables:

1. Shared Data to Cacheline Table

This section lists cache lines sorted by the total number of Remote and Local HITM events:

Cacheline Address	Rmt HITM	Lcl HITM	Total Loads	Primary Allocating Symbol / Structure
0x00007f9a12c40040	1,420	5,120	450,210	g_worker_stats (worker.c:42)
0x00007f9a12c40080	12	45	120,400	ring_buffer_tail (queue.c:118)

A high count in the Rmt HITM (Remote Hit Modified) column indicates that multiple CPU cores across different NUMA nodes are violently contending over the same 64-byte memory slice.

Drilling into Specific Cache Line Byte Offsets

In the terminal report, find the cache line exhibiting the highest Remote HITM count and inspect its internal byte breakdown:

# Inspect specific cache line details in interactive mode
sudo perf c2c report -c 0x00007f9a12c40040

The detailed view reveals the exact byte offsets within the 64-byte block where conflicts occur:

• Offset 0x00 (Bytes 0–7): Thread 0 repeatedly updates counter_processed.
• Offset 0x08 (Bytes 8–15): Thread 1 repeatedly updates counter_failed.

Because both counters are adjacent inside the same structure (struct worker_stats), writing to counter_processed continuously invalidates the cache line for the core updating counter_failed, resulting in massive false sharing stalls.

Remediating False Sharing in Source Code

Once perf c2c identifies the contending data structure, developers eliminate false sharing by padding or aligning variables to separate hardware cache lines:

// Contended structure (Vulnerable to False Sharing)
struct worker_stats {
    uint64_t counter_processed; // Resides in same 64-byte line
    uint64_t counter_failed;    // Resides in same 64-byte line
};

// Remediated structure using C11 / C++11 memory alignment
#include <stdalign.h>

struct alignas(64) worker_stats {
    uint64_t counter_processed;
    char pad1[56]; // Pad to 64 bytes
    uint64_t counter_failed;
    char pad2[56]; // Pad to 64 bytes
};

Alternatively, using compiler directives:

// GCC / Clang attribute alignment
struct worker_stats {
    uint64_t counter_processed __attribute__((aligned(64)));
    uint64_t counter_failed    __attribute__((aligned(64)));
};

Verifying Performance Gains Post-Fix

Re-compile the application with cache line padding applied and execute a fresh profiling run:

# Re-run perf c2c to confirm elimination of HITM events
sudo perf c2c record -- ./worker_node --threads 16
sudo perf c2c report --stdio

The report will show Remote HITM dropping to near zero, accompanied by a dramatic reduction in CPU clock cycles per instruction (CPI) and increased processing throughput.

Using perf c2c in Linux equips performance engineers with the precision tools required to diagnose microarchitectural memory contention and unleash true multi-core scalability.