How to Inspect and Debug Real-Time Kernel Preemption Latency in Linux Using cyclictest

Determinism in Linux operating systems depends on the kernel’s ability to service hardware interrupts and schedule real-time threads without experiencing scheduling latency spikes. When running audio processing workstations, industrial automation systems, robotics controllers, or high-frequency telemetry daemons, kernel preemption delays can trigger buffer underruns, packet loss, or missed task deadlines. The cyclictest utility, part of the standard rt-tests suite, provides the industry-standard benchmark for measuring preemption latency under heavy system load. By repeatedly measuring the delta between a thread’s requested wake-up time and its actual execution time, administrators can audit scheduling responsiveness and pinpoint kernel configuration bottlenecks.

Understanding Linux Kernel Preemption and Latency Metrics

Linux processes run either in user mode or kernel mode. Under standard non-preemptible kernels (such as CONFIG_PREEMPT_NONE) or server-oriented dynamic preemption (CONFIG_PREEMPT_VOLUNTARY), a task executing a kernel system call cannot be interrupted until it reaches an explicit rescheduling point or yields control. In low-latency kernels (CONFIG_PREEMPT or CONFIG_PREEMPT_RT), kernel code is preemptible, converting most spinlocks into sleepable rt-mutexes and moving interrupt handlers into dedicated real-time kernel threads.

Preemption latency consists of three distinct phases:

• Hardware Interrupt Latency: The time required for the processor to acknowledge an interrupt signal and begin executing the top-half interrupt handler.
• Interrupt Handler Latency: The duration during which hardware or software interrupts are masked (interrupts disabled via CLI instructions), preventing other threads from waking up.
• Scheduler Latency: The delay between the kernel marking a higher-priority real-time thread as TASK_RUNNING and the CPU performing the context switch to execute the thread’s instructions.

Installing the rt-tests Diagnostic Suite

The cyclictest binary is packaged within the rt-tests repository across major Linux distributions. Install the required debugging and benchmarking packages via the package manager:

# On Debian, Ubuntu, and Debian-derived systems
sudo apt update && sudo apt install -y rt-tests numactl stress-ng

# On Fedora, Rocky Linux, and RHEL-derived distributions
sudo dnf install -y rt-tests numactl stress-ng

# On Arch Linux
sudo pacman -Sy rt-tests numactl stress-ng

Running Baseline Preemption Latency Benchmarks

To acquire an accurate baseline, execute cyclictest using high-resolution POSIX timers (clock_nanosleep), assigning highest real-time FIFO scheduling priority (SCHED_FIFO 99) across all logical processor cores.

# Run cyclictest across all online CPU cores for 100,000 iterations
sudo cyclictest --smp --priority=99 --interval=1000 --distance=0 --loops=100000 --nanosleep

The command flags configure the following measurement environment:

• --smp or -t: Spawns one benchmark thread per active logical CPU core, pinning each thread to its respective core.
• --priority=99 or -p 99: Assigns real-time priority 99 using the POSIX SCHED_FIFO scheduler policy, ensuring standard user tasks cannot preempt the benchmark.
• --interval=1000 or -i 1000: Sets the base sleep interval to 1000 microseconds (1 millisecond).
• --distance=0 or -d 0: Prevents thread sleep offsets, causing all threads to wake simultaneously to test CPU contention.
• --nanosleep or -n: Forces the use of the kernel’s high-resolution timer (HRT) subsystem via clock_nanosleep().

Interpreting cyclictest Terminal Output

During and upon completion of the test, cyclictest displays real-time execution statistics for each measured core:

# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 0.12 0.08 0.05 1/412 18920

T: 0 (18915) P:99 I:1000 C: 100000 Min:      2 Act:    4 Avg:    5 Max:      18
T: 1 (18916) P:99 I:1000 C: 100000 Min:      2 Act:    5 Avg:    5 Max:      22
T: 2 (18917) P:99 I:1000 C: 100000 Min:      3 Act:    4 Avg:    6 Max:      19
T: 3 (18918) P:99 I:1000 C: 100000 Min:      2 Act:    6 Avg:    5 Max:      25

The columns represent the operational diagnostic metrics:

• T: 0 (18915): Thread index 0 and its assigned Linux process identifier (PID).
• P:99: Real-time scheduling priority (99 indicates maximum priority).
• I:1000: Configured wake-up interval in microseconds.
• C: 100000: Number of completed measurement cycles.
• Min: The absolute minimum latency recorded (in microseconds).
• Act: The latency recorded on the most recent timer iteration.
• Avg: The cumulative average scheduling latency.
• Max: The worst-case scheduling latency spike recorded during the run. In mission-critical environments, the Max value is the defining metric.

Generating Latency Histograms for Visual Analysis

Averages can obscure intermittent spikes caused by interrupt handling or CPU power state transitions. To evaluate latency distribution, generate a histogram log by specifying the --histogram parameter:

# Run benchmark for 5 minutes and output a latency histogram up to 100us
sudo cyclictest --smp -p 99 -i 1000 -d 0 -l 300000 --nanosleep --histogram=100 > /tmp/cyclictest_histogram.dat

Inspect the generated histogram data to locate percentile distribution patterns:

# Inspect the top 20 lines of the generated histogram
head -n 25 /tmp/cyclictest_histogram.dat

In well-tuned real-time systems, 99.9% of all timer samples fall below 15 microseconds, with no long-tail outliers exceeding 30 to 50 microseconds.

Simulating Extreme Workloads to Uncover Latency Spikes

Testing an idle machine rarely reveals real-world scheduling regressions. To identify kernel lock contention and driver latency issues, run cyclictest concurrently with synthetic CPU, virtual memory, and disk I/O stress tests using stress-ng:

# Terminal 1: Launch background CPU, memory, and IO stress workers
stress-ng --cpu $(nproc) --io 4 --vm 2 --vm-bytes 2G --fork 4 --timeout 300s &

# Terminal 2: Run cyclictest with trace break enabled
sudo cyclictest --smp -p 99 -i 1000 -d 0 -l 300000 --nanosleep --breaktrace=100

The --breaktrace=100 flag instructs cyclictest to immediately freeze execution and stop the kernel ftrace tracer if any single thread experiences latency exceeding 100 microseconds. This allows engineers to inspect the exact kernel trace buffer in /sys/kernel/debug/tracing/trace to identify which device driver or spinlock blocked the scheduler.

Mitigating High Preemption Latencies in Linux

When tests reveal undesirable latency spikes, apply the following kernel and hardware optimisations:

• Disable CPU Frequency Scaling and C-States: Deep processor sleep states (C3/C6) introduce wake-up latency when exiting low-power modes. Lock the CPU to maximum performance via cpupower frequency-set -g performance.
• Prevent Deep Idle Transitions: Append processor.max_cstate=1 intel_idle.max_cstate=0 idle=poll to the kernel boot parameters in /etc/default/grub.
• Disable SMT / Hyper-Threading: Simultaneous Multi-Threading allows adjacent logical threads to compete for CPU L1/L2 cache and execution units. In dedicated real-time environments, disable SMT in BIOS or via echo off > /sys/devices/system/cpu/smt/control.
• Isolate Real-Time Cores: Use the isolcpus and nohz_full boot parameters (e.g. isolcpus=2,3 nohz_full=2,3 rcu_nocbs=2,3) to prevent standard kernel housekeeping threads and timer ticks from interrupting target real-time cores.