How to Trace CPU Scheduler Run Queue Latency in Linux Using runqlat

In high-throughput Linux server environments, unexplained application tail latency, degraded request response times, and intermittent micro-stalls are frequently caused by CPU starvation. When multiple active processes or threads demand compute cycles simultaneously, threads transitioning into the runnable state (TASK_RUNNING) must wait inside the Linux Completely Fair Scheduler (CFS) or EEVDF (Earliest Eligible Virtual Deadline First) run queues until a CPU core becomes available. While traditional monitoring utilities such as top, htop, or vmstat report coarse average CPU utilization percentages, they fail to reveal the actual queuing delay experienced by waiting threads. By leveraging eBPF (Extended Berkeley Packet Filter) through the runqlat utility, systems performance engineers can record exact scheduler queue latency distributions as high-resolution histograms.

Understanding Scheduler Run Queue Latency and CPU Saturation

CPU utilization measures the fraction of time a processor spends executing non-idle instructions. However, a server operating at 85% CPU utilization may either be processing tasks smoothly or experiencing severe thread queueing due to bursty workloads, improper thread affinity (CPU pinning), or suboptimal NUMA node placement. Conversely, a multi-tenant virtual machine reporting moderate CPU usage might suffer substantial scheduler latency due to hypervisor-level CPU overcommit (CPU steal time).

Scheduler run queue latency—also referred to as scheduler wait time—measures the exact duration between when a thread becomes runnable (e.g., after waking from a sleep state, completing disk I/O, or receiving incoming network socket data) and when the operating system kernel actually context switches the thread onto an active CPU core. If this latency stretches into milliseconds, interactive applications and low-latency databases experience severe performance degradation. The runqlat tool captures this metric directly from kernel scheduler tracepoints.

Installing the BCC eBPF Toolchain

The runqlat utility is part of the BPF Compiler Collection (BCC) open-source toolkit. It requires Linux kernel development headers and the BCC userspace packages.

On Debian and Ubuntu systems, install the necessary packages using apt:

sudo apt update
sudo apt install -y bpfcc-tools linux-headers-$(uname -r)

On RHEL, Rocky Linux, or Fedora installations, install the tools via dnf:

sudo dnf install -y bcc-tools kernel-devel-$(uname -r)

Confirm the path of the installed tool. On Debian/Ubuntu distributions, BCC executables are typically located in /usr/sbin/ with a -bpfcc suffix:

which runqlat || which runqlat-bpfcc || ls -l /usr/share/bcc/tools/runqlat

Tracing Global Scheduler Latency Distributions

To measure scheduler wait times across all system threads, execute runqlat with root privileges. By default, it aggregates measurements across all CPU cores and outputs an ASCII power-of-2 histogram upon termination via Ctrl + C or after a specified interval:

sudo /usr/sbin/runqlat-bpfcc 5 1

This command records scheduler events for 5 seconds and prints 1 summary interval:

Tracing run queue latency... Hit Ctrl-C to end.

     usecs               : count     distribution
         0 -> 1          : 14210    |****************************************|
         2 -> 3          : 6842     |*******************                     |
         4 -> 7          : 2105     |*****                                   |
         8 -> 15         : 412      |*                                       |
        16 -> 31         : 85       |                                        |
        32 -> 63         : 18       |                                        |
        64 -> 127        : 3        |                                        |
       128 -> 255        : 0        |                                        |

In a healthy, well-provisioned system, the vast majority of latency values reside in the 0 to 15 microsecond range. If you observe significant bucket counts above 1,000 microseconds (1 millisecond) or extending into hundreds of milliseconds, your system is suffering from chronic CPU saturation.

Reporting Latency in Milliseconds for Saturated Environments

On heavily overloaded nodes or virtualised cloud instances where wait times regularly stretch into multi-millisecond scales, reading microsecond histograms can produce elongated tables. Use the -m flag to convert the histogram scale to milliseconds:

sudo /usr/sbin/runqlat-bpfcc -m 10 1

The resulting histogram buckets are formatted directly in milliseconds:

     msecs               : count     distribution
         0 -> 1          : 45210    |****************************************|
         2 -> 3          : 1204     |*                                       |
         4 -> 7          : 320      |                                        |
         8 -> 15         : 42       |                                        |

Tracing Latency for Specific Process IDs

When profiling an isolated database engine (such as PostgreSQL or MySQL) or a microservice running alongside background batch jobs, you can isolate measurements to a single process hierarchy using the -p option:

sudo /usr/sbin/runqlat-bpfcc -p $(pgrep -f "postgres") 5 1

This filters out all unrelated kernel threads and system processes, allowing developers to assess whether query execution lag stems from internal lock contention or external scheduler starvation.

Breaking Down Latency Per Individual CPU Core

Unbalanced CPU core allocation—often caused by incorrect taskset affinities, uneven interrupt distribution (IRQ affinity), or misconfigured cgroups—can leave some CPU cores idle while others experience severe run queue backlogs.

Use the -C option to output individual latency histograms per physical/logical CPU core:

sudo /usr/sbin/runqlat-bpfcc -C 5 1

Examine the output to identify whether a specific CPU core (e.g., CPU 0 handling all network software interrupts) exhibits abnormal wait times compared to other worker cores.

Tracing Latency by Process Execution Name

To identify which specific applications and executable commands suffer the highest scheduler delays, pass the -P flag to generate distinct histograms per process name (comm):

sudo /usr/sbin/runqlat-bpfcc -P 10 1

This displays individual breakdown tables for processes such as nginx, redis-server, and python3, immediately highlighting which services endure thread queuing delays.

Adding Timestamps for Periodic Monitoring

To integrate scheduler queue tracing into continuous operational logging, pass the -T option to prefix each report interval with standard wall-clock timestamps:

sudo /usr/sbin/runqlat-bpfcc -T 60

This generates timestamped 60-second summary histograms, enabling systems administrators to correlate historical latency spikes with scheduled cron routines, garbage collection cycles, or sudden user traffic surges.