How to Disassemble Binary Code and Inspect CPU Cycles in Linux Using perf annotate

When profiling high-performance applications, identifying that a specific function consumes 40% of total CPU cycles is often only the starting point of performance engineering. High-level profiling tools like perf top and perf report aggregate metrics at the symbol or function boundary, but optimizing algorithmic inner loops or low-level systems code requires knowing precisely which assembly instructions cause processor pipeline stalls, branch mispredictions, or memory cache misses.

The Linux perf annotate sub-command provides line-by-line machine code disassembly interleaved with live hardware Performance Monitoring Unit (PMU) sample percentages. By mapping sampled CPU cycles directly onto disassembled x86-64 or ARM64 instructions, developers can pinpoint memory load stalls, vectorization inefficiencies, and branch penalties at the assembly instruction level.

Prerequisites for Instruction-Level Binary Annotation

To accurately correlate instruction pointers (IP) with source code lines and symbol names, ensure the target binary and system environment meet the following requirements:

• Debug Symbols: Compile the target binary with debug information enabled (using -g or -gline-tables-only with GCC or Clang). Avoid stripping symbol tables.
• Disassembler Tooling: perf annotate relies on an underlying disassembler engine, typically GNU objdump or LLVM llvm-objdump. Ensure binutils is installed on your Linux distribution.
• Kernel Permissions: Profiling kernel functions or multi-tenant system processes requires root privileges or relaxed perf event permissions (sysctl kernel.perf_event_paranoid=0 or -1).

To install binutils and the Linux performance tools on Ubuntu or Debian systems, execute:

sudo apt update && sudo apt install linux-tools-common linux-tools-$(uname -r) binutils

Recording Instruction Samples with perf record

Before running perf annotate, collect hardware performance samples using perf record. To capture accurate instruction pointers, sample hardware CPU cycles or instructions at high resolution.

To record CPU cycle samples for a specific application execution, run:

perf record -e cycles -F 999 -- ./my_workload_binary

If you are profiling a long-running process or background daemon, attach to its process identifier (PID):

sudo perf record -e cycles -p 4821 -- sleep 10

To ensure precise instruction attribution without processor skid (where out-of-order execution attributes PMU interrupts to adjacent instructions), use precise event modifiers if supported by your CPU (such as Intel PEBS or AMD IBS):

perf record -e cycles:ppp -- ./my_workload_binary

This writes raw execution event samples to perf.data in your current working directory.

Launching perf annotate in Text and Interactive TUI Modes

Once the sample data is recorded, use perf annotate to disassemble the hottest functions.

To view an ASCII disassembly dump directly in standard output, run:

perf annotate --stdio

By default, the --stdio flag outputs disassembled assembly code for the most heavily sampled function in the recording. Each instruction is prefixed with the percentage of total captured samples that landed on that specific instruction pointer.

To open an interactive Text User Interface (TUI) that allows you to browse symbols, jump across branch targets, and toggle source code interleaving, execute:

perf annotate

Within the interactive TUI interface:

• Use Up / Down Arrow keys to scroll through instructions.
• Press Tab to cycle between hotspots (lines with high sample density).
• Press Enter on a branch or jump instruction (such as jne or callq) to navigate directly to the target label.
• Press H to access the keyboard shortcuts help menu.
• Press Q or Esc to return to the symbol overview or exit.

Inspecting Interleaved Source Code and Assembly

If your binary was compiled with debug flags (-g) and the original source files are accessible, perf annotate automatically interleaves high-level C/C++ or Rust source lines above the corresponding blocks of assembly instructions.

To explicitly enforce source code line interleaving in standard output mode, use:

perf annotate --source --stdio

A representative snippet appears as follows:

         :  for (size_t i = 0; i < count; ++i) {
         :      total += data[i] * multiplier;
    0.12 :    401214:   movslq %ecx,%rax
   72.48 :    401217:   imull  (%rdi,%rax,4),%edx
    0.05 :    40121b:   addl   %edx,%esi
   24.15 :    40121d:   addl   $0x1,%ecx
    3.20 :    401220:   cmpl   %r8d,%ecx
         :    401223:   jne    401214

In this output, instruction 401217 accounts for 72.48% of all captured CPU samples. The instruction dereferences memory ((%rdi,%rax,4)) to multiply an array element. This massive cycle concentration indicates that the processor is stalling on a cache miss waiting for memory retrieval rather than spending time on arithmetic execution.

Targeting Specific Functions and Demangled Symbols

In large enterprise software suites, browsing the entire binary can be cumbersome. You can instruct perf annotate to isolate a specific function symbol directly.

To annotate a specific function name, specify the -s or --symbol flag:

perf annotate -s calculate_matrix_checksum --stdio

For C++ or Rust binaries containing heavily mangled symbol names, perf annotate automatically demangles symbols when binutils is present. If you need to verify raw symbol names or offset offsets, toggle demangling off or on:

perf annotate --demangle -s "MyNamespace::VectorProcessor::Execute" --stdio

To inspect raw machine hexadecimal instruction opcodes alongside the mnemonic assembly instructions, add the --objdump custom flag:

perf annotate --objdump="objdump -M intel" --stdio

This allows switching from standard AT&T syntax to Intel syntax for developers more accustomed to Intel disassembly formats.

Correlating Multiple Hardware Events (L1 Cache Misses and Branch Penalties)

To understand why an instruction consumes excessive cycles, record multiple complementary hardware PMU counters in a single recording session.

To sample both CPU cycles and L1 data cache load misses simultaneously, run:

perf record -e cycles,L1-dcache-load-misses -c 100000 -- ./my_workload_binary

When running perf annotate on a multi-event profile, the tool displays side-by-side percentage columns for each counter:

  Cycles%   L1-Miss%  Disassembly
    0.00      0.00    movslq %ecx,%rax
   81.20     94.50    imull  (%rdi,%rax,4),%edx
    0.10      0.00    addl   %edx,%esi

Here, the high cycle percentage (81.20%) correlates directly with 94.50% of all L1 cache misses occurring at the exact same instruction. This definitively confirms that memory latency, rather than computational complexity, is the primary performance bottleneck, guiding the developer to implement cache-aware blocking, prefetching, or structure-of-arrays memory layouts.