How to Compare Performance Profiles and Detect Regressions in Linux Using perf diff

In continuous performance engineering, software benchmarking, and Linux kernel development, verifying the impact of code modifications requires rigorous differential analysis. When an engineer modifies an algorithm, adjusts compiler optimization flags, or upgrades the kernel from one major release to the next, measuring whether the change improved throughput or introduced an unexpected CPU regression is challenging if relying solely on high-level wall-clock execution timers. Two independent profiling runs rarely capture identical execution timelines or absolute sample counts.

To eliminate manual guesswork and mathematically quantify architectural performance shifts between separate execution runs, the Linux kernel performance tracing suite provides perf diff. By processing two or more recorded profile data files (typically a perf.data baseline against a perf.data.old test run), perf diff computes relative differential percentages across all function symbols and hardware events. It immediately highlights functions that gained overhead (regressions) or shed processor cycles (optimizations). This guide explains how perf diff operates, how to capture reproducible baseline and comparison profiles, and how to analyze differential reports to catch performance regressions in Linux.

Architecture of Differential Performance Profiling with perf diff

The perf diff utility operates by correlating symbol tables and event sample distributions across separate profile recordings:

• Baseline vs Comparison Data: The baseline file represents the reference state (e.g. unoptimized production code or an older kernel build), while subsequent data files capture modified execution states. By default, perf diff compares perf.data.old (baseline) against perf.data (current).
• Differential Delta Calculation: Rather than comparing raw event counts (which fluctuate based on CPU clock scaling, run durations, or system load), perf diff calculates the relative overhead percentage of each function symbol within each run. The differential report presents the mathematical delta (e.g. +3.45% indicates a function consumed 3.45% more of total execution time in the new profile).
• Alternative Comparison Formulas: In addition to basic overhead delta (delta), perf diff supports relative ratio comparisons (ratio) and weighted statistical displacement (wdiff), which highlights functions with substantial sample volume over low-frequency statistical noise.

Using perf diff enables automated performance gate checks in CI/CD pipelines, flagging code commits that inadvertently introduce hot spots into critical execution paths.

Verifying perf Installation and Environment Prerequisites

Ensure that the Linux kernel performance tooling is installed and matches your running kernel release:

# Ubuntu / Debian
sudo apt update
sudo apt install -y linux-tools-common linux-tools-generic linux-tools-$(uname -r)

# RHEL / Rocky Linux / Fedora
sudo dnf install -y perf

To capture user-space and kernel function symbols with full call stack resolution, compile target binaries with frame pointers (-fno-omit-frame-pointer) and debugging symbols (-g):

gcc -O2 -g -fno-omit-frame-pointer -o benchmark_workload workload.c

Recording Baseline and Comparison Performance Profiles

Differential profiling requires capturing two distinct execution runs using identical sampling frequencies and event definitions.

Step 1: Capturing the Baseline Profile

Record a 10-second sampling profile of your baseline workload or unmodified production binary using perf record. Output the data to a dedicated baseline file via -o:

sudo perf record -F 99 -g -o perf.baseline.data -- ./benchmark_workload --threads 8 --iterations 1000000

Key parameters include:

• -F 99: Samples at 99 Hertz to avoid lockstep synchronization with system timer interrupts.
• -g: Captures full call-graph stack unwinding.
• -o perf.baseline.data: Saves the binary trace to an explicitly named baseline file.

Step 2: Capturing the Modified / Optimized Profile

Apply your code patch, configuration tuning, or kernel parameter adjustments, and record a second profile using the exact same sampling parameters:

sudo perf record -F 99 -g -o perf.optimized.data -- ./benchmark_workload --threads 8 --iterations 1000000

Executing Differential Comparisons with perf diff

Once both binary profile datasets are generated, execute perf diff specifying the baseline followed by the comparison dataset:

perf diff perf.baseline.data perf.optimized.data

The terminal outputs a comparative tabular report:

# Baseline  Delta Overhead  Shared Object       Symbol
# ........  ..............  ..................  .....................................
#
    18.42%          -6.12%  benchmark_workload  [.] calculate_matrix_multiplication
    12.10%          +4.85%  benchmark_workload  [.] allocate_dynamic_buffers
     8.15%          -2.30%  libc.so.6           [.] __memcpy_avx_unaligned
     4.20%          +3.10%  [kernel.kallsyms]   [k] __alloc_pages
     2.10%          -0.50%  benchmark_workload  [.] parse_input_tokens

Analyzing this differential report reveals critical performance insights:

• calculate_matrix_multiplication (Delta -6.12%): Demonstrates an optimization success. The function’s total CPU footprint dropped from 18.42% down to 12.30%, indicating the algorithmic patch reduced execution overhead.
• allocate_dynamic_buffers (Delta +4.85%): Flags an unintended regression. The new code is spending significantly more cycles in memory buffer management, which also increased kernel page allocation overhead (__alloc_pages increased by +3.10%).

Advanced Differential Comparison Modes (Ratio and Weighted Diff)

Depending on the analytical objective, standard linear deltas can obscure important changes in low-overhead functions or overemphasize high-frequency statistical jitter.

Computing Relative Ratios

To view relative speedup or slowdown ratios rather than percentage point deltas, use the --compute ratio parameter:

perf diff --compute ratio perf.baseline.data perf.optimized.data

Output columns display ratios (e.g. 0.50 signifies a 2x reduction in overhead, while 1.50 indicates a 50% increase in CPU time):

# Baseline      Ratio  Shared Object       Symbol
# ........  .........  ..................  ................................
    18.42%       0.67  benchmark_workload  [.] calculate_matrix_multiplication
    12.10%       1.40  benchmark_workload  [.] allocate_dynamic_buffers

Highlighting High-Confidence Regressions with Weighted Diff (wdiff)

When analyzing large production datasets, low-frequency functions may show erratic deltas due to sampling noise. The wdiff computation weights deltas by the overall sample volume:

perf diff --compute wdiff:1,1 perf.baseline.data perf.optimized.data

Functions appearing at the top of the wdiff report represent genuine, high-volume architectural regressions that justify developer investigation.

Filtering and Sorting Differential Reports

To streamline automated reporting and focus on specific binary modules:

# Filter output to show only symbols from your target application binary
perf diff --dso benchmark_workload perf.baseline.data perf.optimized.data

# Sort output by descending differential delta (highest regressions first)
perf diff -s delta perf.baseline.data perf.optimized.data

By integrating perf diff into automated continuous benchmarking harnesses, engineering teams can catch CPU regressions immediately during code review, ensuring software releases remain fast and resource-efficient.