How to Benchmark Kernel Subsystems and System Hardware in Linux Using perf bench

Low-level Linux performance tuning, kernel development, and hardware validation require reproducible benchmarks that isolate specific subsystem interactions rather than relying on high-level synthetic scores. While standalone benchmarking suites such as sysbench, fio, or lmbench provide valuable multi-component assessments, the official Linux kernel performance tracing suite includes a dedicated, highly integrated benchmarking framework: perf bench. Built directly into the kernel source tree under tools/perf, perf bench assesses low-level scheduler contention, IPC latency, virtual memory copy operations, epoll scalability, NUMA page migration, and syscall execution speeds without external runtime dependencies.

Because perf bench integrates directly with hardware performance counters and kernel tracepoints, systems engineers can run microbenchmarks while concurrently sampling CPU cache misses, branch mispredictions, context switches, and TLB invalidations. This guide explains how to install prerequisites, navigate available benchmark subsystems, execute targeted microbenchmarks across scheduler, memory, IPC, and NUMA workloads, and interpret numerical results to isolate kernel bottlenecks.

Verifying perf Installation and Viewing Available Subsystems

The perf bench utility is part of the standard Linux linux-tools-common package. Because perf must match or closely track your running kernel release, ensure the architecture-matched kernel tooling packages are installed before running benchmarks.

On Debian and Ubuntu systems, install the kernel performance suite using apt:

sudo apt update
sudo apt install -y linux-tools-common linux-tools-generic linux-tools-$(uname -r)

On enterprise RHEL, Rocky Linux, or Fedora distributions, install perf using dnf:

sudo dnf install -y perf

To view all built-in benchmark subsystems and test collections supported by your running kernel build, execute perf bench with the all parameter set to list:

perf bench all

The output displays functional testing suites categorised under core kernel subsystems:

# Running all benchmarks ...
# Subsystem: sched
Suite: messaging - Benchmark for scheduling and IPC
Suite: pipe      - Benchmark for pipe() latency and throughput

# Subsystem: mem
Suite: memcpy    - Benchmark for memcpy() implementations
Suite: memset    - Benchmark for memset() implementations

# Subsystem: numa
Suite: mem       - Benchmark for NUMA memory access patterns

# Subsystem: futex
Suite: hash      - Benchmark for futex hash table contention
Suite: wake      - Benchmark for futex wake latency
Suite: requeue   - Benchmark for futex requeue operations

# Subsystem: epoll
Suite: wait      - Benchmark for epoll_wait event dispatcher
Suite: ctl       - Benchmark for epoll_ctl registration overhead

# Subsystem: internals
Suite: synthesize - Benchmark for synthetic perf event processing

To inspect options specific to any individual subsystem or suite, append --help to the suite name:

perf bench sched messaging --help
perf bench mem memcpy --help
perf bench futex wake --help

Benchmarking Kernel Scheduler and IPC Latency

The sched subsystem within perf bench evaluates process creation, context switching overhead, thread wakeup latency, and inter-process communication (IPC) efficiency across local sockets and Unix pipes.

Benchmarking Scheduler Contention with sched messaging

The sched messaging suite replicates the classic hackbench workload. It launches groups of sender and receiver threads or processes that pass small data packets back and forth through Unix domain sockets or pipes, saturating the Linux Completely Fair Scheduler (CFS / EEVDF) with rapid context switches:

perf bench sched messaging

By default, perf bench sched messaging executes 10 groups of 20 sender and receiver processes transferring 100 100-byte messages. The output displays the elapsed wall-clock time required to complete all message transactions:

# Running 'sched/messaging' benchmark:
# 20 sender and receiver processes per group
# 10 groups == 400 processes run

     Total time: 0.124 [sec]

To stress multi-socket NUMA servers or high-core processors, increase the number of process groups and switch between thread mode (--threads) and separate process address spaces (default):

# Run 32 process groups across pthreads using Unix pipes instead of sockets
perf bench sched messaging --groups 32 --threads --pipe

Lower execution time indicates superior scheduler throughput and reduced context-switching overhead under multi-threaded lock contention.

Measuring Context Switch Round-Trip Latency with sched pipe

While messaging stresses multi-process queue saturation, sched pipe measures single-threaded round-trip latency between two cooperative tasks communicating across a unidirectional Unix pipe:

perf bench sched pipe --loop 1000000

The test executes 1,000,000 round-trip token transfers and reports the total elapsed execution duration and the exact average latency per operational round-trip:

# Running 'sched/pipe' benchmark:
# Executed 1000000 pipe operations between two processes

     Total time: 1.482 [sec]

       1.482000 usecs/op
         674763 ops/sec

This benchmark reveals the real-world microsecond latency of thread descheduling, kernel pipe buffer synchronisation, and processor cache warming.

Benchmarking Memory Bandwidth and Routine Throughput

The mem subsystem assesses memory subsystem throughput, specifically evaluating how fast the CPU and memory controller copy and initialise contiguous memory blocks using standard C runtime functions like memcpy() and memset().

Benchmarking memcpy Performance Across Buffer Sizes

Memory bandwidth varies significantly depending on whether target data resides in L1 data cache (32–48 KB), L2 cache (1–2 MB), shared L3 cache (16–128 MB), or physical system RAM. The mem memcpy benchmark allows testing specific byte lengths and memory transfer implementations (e.g. x86-64 SIMD registers like AVX2 or rep_movsq):

# Benchmark a 64MB buffer copy to stress physical DDR4/DDR5 system memory
perf bench mem memcpy --size 64MB --iterations 20

The test reports data throughput in gigabytes per second (GB/sec):

# Running 'mem/memcpy' benchmark:
# Copying 64MB bytes ...

       28.431289 GB/sec (average)

To test cache-line performance rather than main memory bus saturations, test small buffer sizes that fit cleanly inside L1 or L2 CPU caches:

# Benchmark 32KB buffer copy inside CPU L1 cache
perf bench mem memcpy --size 32KB --iterations 100000

Comparing Alternative memcpy Architectures

Modern Linux kernels dynamically select string and memory copy algorithms based on CPU capability flags. You can compare the throughput of standard glibc routines against custom kernel assembly variants using the --routine flag:

# List available routine implementations
perf bench mem memcpy --routine help

# Benchmark x86-64 rep_movsq hardware string instructions
perf bench mem memcpy --size 16MB --routine x86-64-rep-movsq

# Benchmark default glibc vectorized assembly
perf bench mem memcpy --size 16MB --routine default

Benchmarking Multi-Threaded Futex Lock Contention

Fast User-space Mutexes (futexes) provide the foundational locking and condition variable primitives used by POSIX threads (pthread_mutex, pthread_rwlock). The perf bench futex suite quantifies kernel lock hash table collisions and thread wake-up scheduling delays.

Evaluating Futex Wakeup Latency

When multiple worker threads block waiting for a shared lock, the kernel must traverse internal bucket chains to wake target threads. Execute futex wake to measure wake-up efficiency across multiple contending threads:

perf bench futex wake --threads 16 --parallel 1000

Output illustrates the microsecond latency cost of unlocking parallel futex queues:

# Running 'futex/wake' benchmark:
# Waking 16 threads in 1000 loops

   Woken: 16000
   Total time: 0.082 [sec]
       5.125000 usecs/op

Benchmarking Futex Hash Collisions

Large multi-socket enterprise servers running database engines can encounter kernel lock contention in the global futex hash buckets. Test futex hash table scaling using futex hash:

perf bench futex hash --threads 32 --runtime 10

This benchmark runs 32 concurrent threads creating and destroying futex references for 10 seconds, reporting aggregate operations per second:

# Running 'futex/hash' benchmark:
# Hashing 32 threads for 10 seconds

   Completed 14820942 operations in 10.001 [sec]
   Throughput: 1481946 ops/sec

Benchmarking NUMA Memory Access Latency

In Non-Uniform Memory Access (NUMA) multi-socket servers, accessing memory attached to a local CPU socket is significantly faster than traversing interconnect fabrics (such as AMD Infinity Fabric or Intel Ultra Path Interconnect) to fetch data from remote node memory. The numa mem benchmark quantifies local versus cross-node memory access penalties.

Execute the NUMA memory benchmark across multiple CPU nodes:

perf bench numa mem -p 2 -t 8 -G 4 -l 5 -s 1000

Key parameters configure NUMA topology testing:

• -p 2: Allocates 2 concurrent process groups.
• -t 8: Allocates 8 worker threads per process group.
• -G 4: Uses 4 GB of global shared memory.
• -l 5: Executes 5 measurement iterations.
• -s 1000: Sleeps 1000 milliseconds between execution phases.

The resulting output breaks down system convergence time, remote memory read throughput, and cross-node cache coherency overhead, providing empirical data to tune NUMA balancing parameters like kernel.numa_balancing in sysctl.

Combining perf bench with Hardware Event Profiling

The primary architectural advantage of perf bench over third-party benchmarking tools is that you can prefix any benchmark command with perf stat or perf record to monitor hardware performance counters, cache miss rates, and branch predictions during the test run.

To inspect CPU instructions per cycle (IPC), cache misses, and context switches while running the scheduler messaging benchmark, wrap the command with perf stat:

perf stat -e task-clock,context-switches,cpu-migrations,page-faults,cycles,instructions,cache-references,cache-misses perf bench sched messaging -g 10

The resulting report combines benchmark completion duration with low-level microarchitectural statistics:

# Running 'sched/messaging' benchmark:
# 20 sender and receiver processes per group
# 10 groups == 400 processes run

     Total time: 0.118 [sec]

 Performance counter stats for 'perf bench sched messaging -g 10':

            648.12 msec task-clock                       #    4.812 CPUs utilized
            42,108      context-switches                 #   64.969 K/sec
               284      cpu-migrations                   #  438.190 /sec
             4,912      page-faults                      #    7.579 K/sec
     2,450,182,310      cycles                           #    3.780 GHz
     1,980,412,050      instructions                     #    0.81  insn per cycle
        28,491,200      cache-references                 #   43.960 M/sec
         4,120,490      cache-misses                     #   14.46% of all L1D hits

       0.134689201 seconds time elapsed

By reviewing the instructions per cycle (insn per cycle) and cache-misses percentages alongside execution time, administrators can verify whether kernel parameter adjustments (such as changing scheduler tunables in /proc/sys/kernel/sched_* or modifying transparent huge pages in /sys/kernel/mm/transparent_hugepage/) genuinely reduce processor stalls and improve real-world system throughput.