How to Profile Live Functions and CPU Cycles in Linux Using perf top

When investigating sudden processor saturation or performance degradation on production Linux systems, running static snapshot profilers or capturing multi-gigabyte trace files with perf record can be impractical. System administrators and performance engineers often require an immediate, interactive, real-time view of which kernel functions, system libraries, and application symbols are consuming CPU cycles as workloads fluctuate.

While standard utilities like top or htop report per-process and per-thread CPU percentages, they provide zero insight into internal function execution or kernel space overhead. The Linux performance subsystem provides perf top, a dynamic Text User Interface (TUI) profiler that samples processor hardware Performance Monitoring Unit (PMU) counters continuously in real time. This guide covers how to launch perf top, filter by process or CPU core, inspect kernel versus user-space overhead, and drill down directly into live assembly disassembly.

Architecture of perf top Real-Time Sampling

The perf top utility operates by configuring a high-frequency sampling timer on the target CPU cores via the perf_event_open() system call:

• PMU Event Sampling: By default, perf top samples hardware cycles (CPU clock cycles). At regular intervals (typically 4000 Hz / samples per second), the CPU hardware generates an interrupt, recording the current instruction pointer (IP).
• Symbol Resolution: The tool resolves the captured instruction pointer against runtime symbol tables, including the kernel symbol table (/proc/kallsyms), dynamic shared libraries (via /proc/[pid]/maps and ELF symbols), and user-space binaries.
• Dynamic Decay and Refresh: Similar to standard top, perf top periodically refreshes its display (default: every 2 seconds). It applies a mathematical decay weight to historical samples, ensuring the screen reflects active workload shifts rather than historical averages.

Prerequisites and Kernel Privileges

Sampling system-wide CPU cycles and kernel functions requires elevated privileges or relaxed perf event kernel parameters.

To install the necessary performance tools package on Ubuntu or Debian:

sudo apt update && sudo apt install linux-tools-common linux-tools-$(uname -r)

To inspect or relax kernel profiling permissions without requiring continuous root access:

# Check current paranoid level (default is often 2 or 3)
cat /proc/sys/kernel/perf_event_paranoid

# Temporarily allow non-root users to sample CPU cycles and kernel symbols
sudo sysctl kernel.perf_event_paranoid=0

Launching and Navigating the perf top Interface

To launch real-time system-wide profiling across all running processes and kernel threads, execute:

sudo perf top

The interactive Text User Interface (TUI) initializes immediately, presenting a live ranked view of hot functions:

Samples: 32K of event 'cycles', 4000 Hz, Event count (approx.): 24198210
Overhead  Shared Object       Symbol
  18.42%  [kernel]            [k] copy_user_enhanced_fast_string
  12.10%  my_backend_service  [.] ProcessTransactionQueue
   8.45%  libc.so.6           [.] __memcmp_avx2_movbe
   6.12%  [kernel]            [k] do_syscall_64
   4.25%  [kernel]            [k] finish_task_switch.isra.0

Key UI elements and navigation controls include:

• Overhead: The percentage of captured CPU cycle samples that occurred inside that specific function during the refresh interval.
• Shared Object: Differentiates kernel code ([kernel]), dynamic shared C libraries (libc.so.6), and specific application binaries.
• Symbol Modifiers: A symbol bracketed with [k] indicates kernel-space execution; [.] indicates user-space execution.
• Navigation: Use the Up / Down Arrow keys to navigate between symbols. Press Enter to open the contextual action menu. Press H for keyboard help, and Q to exit.

Filtering Profiling Streams by Process, User, or CPU Core

On busy multi-tenant servers, system-wide profiling can produce noisy outputs. You can target perf top to isolate specific execution scopes.

To profile a specific process identifier (PID):

sudo perf top -p 8412

To monitor only a specific CPU core (for example, diagnosing latency spikes on core 4):

sudo perf top -C 4

To filter strictly for user-space application code, excluding kernel symbols entirely, use the :u event modifier:

sudo perf top -e cycles:u

Conversely, to profile only kernel-space execution when troubleshooting high system CPU load:

sudo perf top -e cycles:k

Interactive Live Assembly Annotation

One of the most effective capabilities of perf top is instantaneous assembly disassembly of live hotspots.

To inspect the machine instructions driving high CPU consumption:

1. Use the arrow keys to highlight the hot symbol in the perf top display.
2. Press Enter to open the menu dialogue.
3. Select Annotate [symbol_name].

The interface switches to a side-by-side assembly disassembly view, displaying live sample percentages next to each assembly instruction (e.g. mov, vmovdqa, cmp, jne). Instructions responsible for CPU pipeline stalls or memory wait states illuminate in red, pinpointing the exact machine instruction causing the bottleneck.

Press Q to return to the live symbol overview.

Customizing Sampling Frequency and Refresh Intervals

You can adjust the sampling rate and display refresh cadence to balance profiling precision with profiling overhead.

To lower the sampling frequency to 1000 Hz to minimize overhead on production systems:

sudo perf top -F 1000

To increase the screen refresh rate to once every 0.5 seconds for catching rapid transient CPU bursts:

sudo perf top -d 0.5

To output a plain text summary to standard output instead of the interactive ncurses TUI (ideal for logging or remote terminal pipes):

sudo perf top --stdio