How to Package Symbol Binaries for Offline Analysis in Linux Using perf archive

When profiling high-load applications or operating system kernels on production Linux servers, performing interactive analysis directly on the target host is often impractical or prohibited by operational policies. Running interactive disassembly tools or debug symbol servers can disrupt live workloads, introduce CPU overhead, or violate security policies that restrict developer access to production environments. The standard workflow involves capturing hardware performance counters and call stacks to a perf.data file using perf record and copying the data file to a local workstation for post-mortem analysis.

However, running perf report or perf annotate on a separate workstation usually fails to resolve symbol names, displaying frustrating [unknown] entries or failing to locate the original binary executables and shared dynamic libraries. Because the target server and the developer workstation often have different library versions, compiler builds, or kernel distributions, the Build IDs embedded in the binaries do not match. The Linux kernel profiling subsystem provides a dedicated utility, perf archive, designed specifically to resolve this disparity by bundling every binary, shared library, and kernel module referenced in a profile into a portable, self-contained archive.

Understanding Build IDs and Symbol Portability

Modern Linux ELF binaries and shared libraries contain a unique SHA-1 or MD5 hash known as a Build ID, embedded inside an ELF note section (.note.gnu.build-id). When perf record samples instruction pointers across user and kernel space, it inspects these ELF headers and records the precise Build ID of every executable file that contributed samples to the profile.

When you transfer only the perf.data file to another computer, perf report reads those recorded Build IDs and searches the local workstation filesystem and local debug directories (such as /usr/lib/debug/.build-id/) for matching binaries. If the workstation runs a slightly different patch version of libc.so.6 or an entirely different kernel build, the local files will have different Build IDs. Consequently, the profiler cannot map memory offsets back to function names, source lines, or machine instructions. By using perf archive, you package the exact ELF binaries that executed on the production machine, guaranteeing perfect symbol resolution and accurate disassembly on any analysis workstation.

Recording Profiles with Build ID Recording Enabled

Before creating an archive, ensure that your profiling session records complete Build ID metadata. By default, perf record records Build IDs at the end of the recording session, but you can explicitly confirm this behaviour or inspect recorded Build IDs using the terminal.

Execute your profiling workload and record performance events across your system or targeted application:

# Record CPU cycles and call stacks for a specific process
sudo perf record -F 99 -g -p 18452 -- sleep 30

# Alternatively, record a system-wide hardware profile
sudo perf record -F 99 -a -g -- sleep 10

Once the session concludes, verify that perf successfully populated the internal Build ID header within the generated perf.data file by inspecting it with perf buildid-list:

# List all binaries and their associated Build IDs captured in the profile
perf buildid-list -i perf.data

The command outputs a list of 40-character hexadecimal strings followed by the absolute file paths of the profiled binaries, dynamic libraries, and kernel images:

e4a19b8cf149c71a34d287bb184e49320b7842a1 /usr/lib/x86_64-linux-gnu/libc.so.6
7f3c21a941ef0833158b45610cc442b324d8520f /usr/bin/nginx
c9811425e4088a101b0f0238816c52a0a256df92 [kernel.kallsyms]
81290ff0e20148719bcad102390ff198762512cd /usr/lib/x86_64-linux-gnu/libssl.so.3

Creating a Portable Symbol Bundle with perf archive

Once you have confirmed that the perf.data file contains recorded Build IDs, invoke the perf archive command in the directory containing the data file:

# Generate a symbol archive from the default perf.data file
sudo perf archive

If your profile was written to a custom filename or specific path, specify the file using the standard input argument or pass the filename directly:

# Package symbols for a specific profile file
sudo perf archive /var/log/profiles/database_bench.perf.data

The perf archive utility examines the Build IDs stored in the profile, locates the physical binaries inside the host root filesystem and system debug repositories (including ~/.debug/), and bundles them into a compressed archive. By default, the command creates a tarball named perf.data.tar.bz2 (or <filename>.tar.bz2) in the current working directory.

You can verify the contents of the generated archive using tar without extracting it:

# Inspect the files inside the generated archive
tar -tvjf perf.data.tar.bz2 | head -n 20

The archive structure mirrors the internal ~/.debug/ hierarchy, organising binaries by their Build ID prefixes (for example, .debug/.build-id/e4/a19b8cf149c71a34d287bb184e49320b7842a1/elf). This exact structure allows the analysis machine to automatically match samples to binaries regardless of where the software was originally installed.

Unpacking and Analysing Profiles on the Workstation

To analyse the recorded profile on your development workstation or forensic environment, transfer both the profile data file and the symbol archive using rsync or scp:

# Transfer data and archive files to the local machine
scp [email protected]:/tmp/perf.data .
scp [email protected]:/tmp/perf.data.tar.bz2 .

On the destination workstation, the symbol cache must be extracted directly into your user home directory or system debug cache so that perf can discover the files during analysis:

# Create the local debug directory if it does not exist
mkdir -p ~/.debug

# Extract the archive into your user debug directory
tar -xvjf perf.data.tar.bz2 -C ~/.debug/

Once the archive is unpacked, launch perf report on the workstation:

# Run report using the transferred profile
perf report -i perf.data

The profiler automatically checks ~/.debug/.build-id/ for matching binaries corresponding to the Build IDs listed in perf.data. Instead of displaying unmapped memory addresses or [unknown] functions, perf report presents complete function symbols, correct DSO library classifications, and fully navigable call graphs.

Verifying Symbol Resolution and Disassembly

To confirm that both symbol definitions and machine instructions are available, drill down into an intensive function within the interactive perf report browser and press a to trigger code annotation, or invoke perf annotate directly from the command line:

# Annotate the hottest function to inspect assembly instructions
perf annotate -i perf.data --stdio --symbol=ngx_http_process_request

If the symbols and binaries were bundled correctly, perf annotate displays the disassembled assembly instructions side by side with the percentage of sampled CPU cycles attributed to each instruction. If source code was compiled with DWARF debug flags (-g) and packaged alongside the binary, line numbers and source lines will also be displayed above the relevant instructions.

Automating Archive Generation in Continuous Profiling Pipelines

In automated profiling pipelines and diagnostic scripts, you can streamline the capture, archiving, and compression workflow using a concise shell script that removes temporary overhead and handles compression cleanly:

#!/usr/bin/env bash
set -euo pipefail

OUTPUT_DIR="/var/profiles/$(date +%Y%m%d_%H%M%S)"
mkdir -p "${OUTPUT_DIR}"

echo "Collecting 20-second system profile..."
perf record -F 99 -a -g -o "${OUTPUT_DIR}/perf.data" -- sleep 20

echo "Bundling symbols and binaries..."
(
    cd "${OUTPUT_DIR}"
    perf archive perf.data
)

echo "Compressing profile bundle..."
tar -czf "${OUTPUT_DIR}_bundle.tar.gz" -C "${OUTPUT_DIR}" perf.data perf.data.tar.bz2
rm -rf "${OUTPUT_DIR}"

echo "Profile bundle generated: ${OUTPUT_DIR}_bundle.tar.gz"

This script ensures that system engineers capture both the runtime samples and the matching binary symbols in a single, transportable package, preventing symbol mismatch issues across distributed development and operations teams.