How to Manage Long-Running Background Profiling Sessions in Linux Using perf daemon

Capturing CPU performance profiles, hardware PMU counter metrics, and tracepoints on production Linux servers typically requires interactive shell sessions using perf record. However, in enterprise environments and long-term diagnostic scenarios, requiring an interactive terminal to remain open while profiling is impractical. Transient terminal disconnections terminate running processes, while managing background jobs with process IDs and shell background operators (&) makes coordinating multiple recording sessions and rotating data buffers error-prone.

To provide robust continuous profiling capabilities, the Linux kernel profiling subsystem includes perf daemon. The daemon utility operates as a persistent background supervisor that coordinates multiple concurrent profiling sessions, controls ring buffer recording, and exposes a clean control interface for taking point-in-time snapshots without terminating the recording daemon.

Architecture of perf daemon

The perf daemon architecture separates session orchestration from performance data collection. Instead of manually invoking disparate perf record commands, administrators define a declarative configuration file that specifies:

• Base Directory: The filesystem root path where the daemon maintains its control socket, lock files, and session output folders.
• Multiple Named Sessions: Distinct profiling profiles running concurrently with independent sampling rates, hardware events, and CPU affinities.
• Ring Buffer Snapshot Mode: Continuous in-memory recording where data is written to a ring buffer and flushed to disk only when an external trigger or signal occurs.
• Control Socket Interface: A local UNIX domain socket through which administrators send signals, query session status, or pause recording.

Creating the perf daemon Configuration File

Before launching the daemon, create a configuration file formatted according to the standard daemon syntax. The file defines global daemon settings followed by individual session blocks.

Create a configuration file at /etc/perf-daemon.conf or within your user workspace:

# Base directory for socket control and session storage
[daemon]
base = /var/log/perf-daemon

# Session 1: Continuous low-overhead CPU cycle sampling
[session-cycles]
run = -m 1M -e cycles -F 49 -a -o /var/log/perf-daemon/cycles.data

# Session 2: Snapshot-based kernel lock contention monitoring
[session-locks]
run = -m 2M -e lock:lock_acquire,lock:lock_acquired --switch-output=100M -a -o /var/log/perf-daemon/locks.data

In this example, session-cycles continuously samples CPU cycles across all CPUs at 49 Hz with a 1 MB memory buffer. The second session, session-locks, monitors kernel lock tracepoints and automatically rotates output files once they reach 100 MB.

Starting and Supervising the Daemon

To start the background daemon process using your configuration file, invoke perf daemon start with elevated privileges:

# Start the background profiling daemon
sudo perf daemon start --config /etc/perf-daemon.conf

The daemon forks into the background, creates its working directories, and initiates the child perf record worker processes. You can verify that the daemon is actively running and managing sessions using the ping command:

# Ping the daemon via its control socket
sudo perf daemon ping --config /etc/perf-daemon.conf

The command returns OK if the control socket responds correctly.

Querying Active Profiling Sessions

To inspect active sessions, CPU usage, process IDs, and output file paths managed by the daemon, run perf daemon without extra arguments or specify the base directory:

# Query active daemon sessions
sudo perf daemon --config /etc/perf-daemon.conf

The output presents a formatted summary of all configured sessions:

[18241:session-cycles] running: /usr/bin/perf record -m 1M -e cycles -F 49 -a -o /var/log/perf-daemon/cycles.data
[18242:session-locks] running: /usr/bin/perf record -m 2M -e lock:lock_acquire,lock:lock_acquired --switch-output=100M -a -o /var/log/perf-daemon/locks.data

Signaling and Snapping Ring Buffers Non-Disruptively

One of the most powerful features of perf daemon is taking instantaneous snapshots of performance data when an application anomaly occurs without stopping the ongoing background trace.

To trigger a snapshot or force an output file rotation, send a signal to the daemon control interface:

# Send snapshot trigger to all active sessions
sudo perf daemon signal --config /etc/perf-daemon.conf

# Alternatively, trigger a snapshot for a specific session
sudo perf daemon signal --session session-cycles --config /etc/perf-daemon.conf

When the signal is received, the target recording process flushes its current memory ring buffer to disk, creates a timestamped perf.data slice, and continues sampling into a fresh buffer.

Stopping Sessions and Analyzing Captured Output

When the diagnostic period ends or you wish to conclude continuous profiling, issue the stop command:

# Gracefully terminate the daemon and all child profiling workers
sudo perf daemon stop --config /etc/perf-daemon.conf

The daemon stops all recording instances, flushes remaining buffer entries to disk, cleans up the control socket, and exits. The resulting data files can now be analysed with standard tools:

# Inspect the captured CPU cycles profile
perf report -i /var/log/perf-daemon/cycles.data --stdio | head -n 30

Using perf daemon ensures that production systems maintain continuous, low-overhead performance observability while allowing engineers to extract precise diagnostic snapshots during latency anomalies.