How to Validate Hardware PMU and Kernel BPF Subsystems in Linux Using perf test

Modern Linux performance profiling relies heavily on CPU Performance Monitoring Units (PMUs), kernel tracepoints, software breakpoint engines, and eBPF bytecode virtual machines. When hardware performance counters misbehave, deliver erratic counter values, or fail due to hypervisor virtualization restrictions, profiling tools such as perf stat, perf record, and perf top produce skewed or incomplete diagnostic metrics. The Linux kernel profiling infrastructure includes a comprehensive automated test suite accessible directly via the perf test command. This built-in harness executes dozens of targeted unit tests across PMU hardware event decoding, sample parsing, dwarf unwinders, memory access traps, and BPF filter execution.

Executing perf test allows system administrators, kernel engineers, and performance analysts to verify whether a server or workstation kernel environment fully supports low-level profiling before initiating complex tracing benchmarks. This guide covers how to run complete and targeted PMU test suites, inspect verbose execution traces, isolate hardware counter discrepancies, and troubleshoot virtualised hypervisor limitations.

Understanding the perf test Automated Diagnostic Architecture

The perf test utility is compiled directly into the perf user-space tool binary and interfaces with the underlying kernel via the perf_event_open() system call and debugfs/tracefs interfaces. Rather than testing application code, perf test evaluates the integrity of the profiling pipeline itself. The test suite divides checks into several functional categories:

• Hardware PMU Counter Verification: Validates architectural cycle counters, retired instructions, branch mispredictions, cache references, and vendor-specific PMU events (such as Intel RAPL, AMD Core/L3 PMUs, or ARM CoreSight).
• Software Events and Tracepoints: Confirms page fault interceptors, task context switches, CPU migration events, and kernel tracepoint parsers.
• Stack Unwinding and Symbol Resolution: Verifies DWARF call frame information (CFI), frame pointer traversal, and libunwind integration across multi-threaded workloads.
• BPF and Subsystem Probing: Compiles and loads transient eBPF bytecode programs to ensure the kernel JIT compiler and tracepoint attachments function without security filter interference.
• Session File Parsing and Synthetic Events: Validates internal file serialization, ring buffer event dispatching, and memory mapping logic.

Listing Available Test Cases and Subtests

Before launching test suites, examine the individual diagnostic checks supported by your installed kernel and perf binary version. Execute perf test -v list or pass the test listing argument:

# Display all available unit test entries with indices
perf test list

The output enumerates individual test numbers alongside descriptive names:

 1: vmlinux symtab matches kallsyms
 2: Detect openat syscall event
 3: Detect openat syscall event on all cpus
 4: Read samples using the read_format
 5: Sample parsing
 6: Parse event definition strings
 7: Simple expression parser
 8: PERF_RECORD_* events & anon char devices
 9: Test dwarf unwind
10: Set breakpoint
...
42: BPF filter
43: Support openat syscall event on all cpus
44: Hardware breakpoints
45: Test branch stack sampling

Notice that some comprehensive checks contain nested subtests. To inspect subtest enumerations for a specific module—such as sample parsing or hardware breakpoints—query the specific test number:

# Inspect subtests for test case 6
perf test -v 6

Running Full Diagnostic Suites Across Subsystems

To perform an exhaustive audit of your system’s profiling capabilities, invoke perf test without arguments. Elevated root privileges are necessary to enable raw PMU hardware programming and tracefs access:

# Execute the full automated perf test suite
sudo perf test

The console displays real-time execution feedback for each test block, categorising outcomes into three primary states:

• Ok: The test executed, verified counter values or ring buffer records, and passed all structural assertions.
• Skip: The system lacks specific hardware registers, running inside a hypervisor with disabled PMU passthrough, or optional development libraries (e.g., libdw or libunwind) were not present at compile time.
• FAILED: The kernel returned an unexpected error code, counter values diverged from expected thresholds, or a memory trap failed.

A typical operational summary reflects clean passes and benign skips for hardware absent on the host platform:

 1: vmlinux symtab matches kallsyms                                 : Ok
 2: Detect openat syscall event                                     : Ok
 3: Detect openat syscall event on all cpus                         : Ok
 4: Read samples using the read_format                              : Ok
 5: Sample parsing                                                  : Ok
 6: Parse event definition strings                                  : Ok
 7: Simple expression parser                                        : Ok
 8: PERF_RECORD_* events & anon char devices                        : Ok
 9: Test dwarf unwind                                               : Ok
10: Set breakpoint                                                  : Ok
11: Hardware breakpoints                                            : Ok
12: Use modify_user_hw_breakpoint to change breakpoint setting      : Ok
13: Breakpoint overflow signal handler test                         : Ok
14: Breakpoint overflow sampling test                              : Ok
15: Number of trigger breakpoints for instruction addresses         : Ok
16: Number of trigger breakpoints for data addresses                : Ok
17: Breakpoint for instruction address and variable length          : Ok
18: Breakpoint for data address and variable length                 : Ok
19: Watchpoint for instruction address and variable length          : Ok
20: Watchpoint for data address and variable length                 : Ok
21: Test branch stack sampling                                      : Ok
22: BPF filter                                                      : Ok

Targeting Specific PMU and Hardware Breakpoint Checks

Running the full test suite can take several minutes. When verifying a newly provisioned virtual machine or a customised kernel build, isolate tests targeting hardware breakpoints, PMU parsing, or BPF execution by index or string pattern.

Validating Hardware Watchpoints and Execution Traps

Hardware debug registers (such as DR0–DR7 on x86_64) provide hardware breakpoints for instruction execution and data access watchpoints without incurring software trapping overhead. Test these registers using:

# Target hardware breakpoint validation suites
sudo perf test "breakpoint"

To inspect the low-level register configuration, add the verbose flag (-v):

# Run breakpoint test with detailed step tracing
sudo perf test -v "Hardware breakpoints"

Verbose output reveals the setup of memory traps and signal dispatching:

--- start ---
test child forked, pid 14892
count 1 bp 00000000a45e9821, target 00000000a45e9821
count 2 bp 00000000a45e9821, target 00000000a45e9821
count 3 bp 00000000a45e9821, target 00000000a45e9821
test child finished with 0
---- end ----
Hardware breakpoints: Ok

Validating Kernel eBPF Filters and JIT Capabilities

When using perf record with custom eBPF filters, the kernel must parse BPF bytecode, verify safety constraints, and attach programs to raw tracepoints. Execute the BPF subsystem verification:

# Run the automated BPF subsystem validation test
sudo perf test -v "BPF"

The verbose log details the compilation and loading of test bytecode into the kernel verifier:

--- start ---
test child forked, pid 15024
Checking for BPF support: [OK]
Compiling test BPF program... [OK]
Attaching test BPF program to tracepoint sys_enter_openat... [OK]
Triggering synthetic workload... [OK]
Verifying event buffer counts... [OK]
test child finished with 0
---- end ----
BPF filter: Ok

Automating PMU Diagnostics in Continuous Integration and Health Checks

In automated provisioning workflows and CI pipelines, validating that performance monitoring counters are active ensures subsequent benchmarking jobs produce reliable results. Use standard shell exit codes to evaluate health status:

#!/usr/bin/env bash
set -euo pipefail

echo "Auditing host performance profiling capabilities..."

# Verify kernel symbol table integrity
if ! sudo perf test "vmlinux symtab matches kallsyms" > /dev/null 2>&1; then
    echo "CRITICAL: Kernel symbols out of sync with /proc/kallsyms. Install linux-tools-$(uname -r)." >&2
    exit 1
fi

# Verify hardware breakpoint and PMU registers
if ! sudo perf test "Hardware breakpoints" > /dev/null 2>&1; then
    echo "WARNING: Hardware breakpoints unavailable. Check hypervisor vPMU configuration." >&2
fi

# Verify BPF filter subsystem
if ! sudo perf test "BPF" > /dev/null 2>&1; then
    echo "NOTICE: BPF filter subsystem not supported or restricted by kernel lockdown." >&2
fi

echo "Performance profiling audit completed successfully."

Troubleshooting Common Test Failures and Skipped States

Understanding the root causes of failing or skipped test cases prevents time spent debugging non-existent software bugs:

• Skipped PMU or Cycle Tests inside Virtual Machines: When running inside AWS EC2, Azure VMs, or VMware ESXi guests, virtual CPUs often run with virtualised PMUs (vPMU) disabled by default. The host hypervisor must be explicitly configured to expose architectural PMU counters (such as enabling vpmucounters=true in VMware or launching bare-metal/vPMU-enabled VM instances in public clouds).
• Kernel Lockdown Restrictions: If Secure Boot is active and kernel lockdown is set to [integrity] or [confidentiality], arbitrary BPF loading, raw MSR access, and kprobe injection will be rejected. Inspect lockdown status using cat /sys/kernel/security/lockdown.
• Missing Debug Symbols (vmlinux): Tests comparing symbol tables against /proc/kallsyms will fail or skip if matching kernel debug packages are not installed. On Debian/Ubuntu systems, install the appropriate package using sudo apt install linux-tools-$(uname -r) linux-tools-generic.
• Paranoid Sysctl Thresholds: If /proc/sys/kernel/perf_event_paranoid is set to 3 or higher, non-root processes cannot access performance counters, and even root execution may face tracefs restrictions. Set the value to 1 or -1 for testing via sudo sysctl -w kernel.perf_event_paranoid=1.

By integrating perf test into standard baseline validation procedures, infrastructure engineers can ensure that hardware counters, kernel tracepoints, and BPF instrumentation are fully functional before diagnostic profiling sessions begin.