How to Trace CPU Scheduler Run Queue Length in Linux Using runqlen

When investigating Linux performance degradation, system administrators often turn to standard system load averages or CPU utilisation metrics. However, high CPU load does not always indicate whether threads are actively computing or stalling in scheduler queues awaiting available core resources. The runqlen utility, provided by the BPF Compiler Collection (BCC), samples the Linux completely fair scheduler (CFS) run queue occupancy across all online CPUs and outputs the distribution as a clear histogram, revealing real-time scheduler saturation.

Understanding Run Queue Occupancy vs Scheduler Latency

In the Linux kernel, every CPU core maintains an independent run queue containing runnable tasks ready to execute. While companion eBPF tools like runqlat measure the exact elapsed time an individual task waits on a run queue before receiving CPU execution slices, runqlen inspects the queue length itself.

Measuring queue length provides distinct analytical advantages:

• Identifying core saturation: Consistent run queue depths greater than one or two tasks per core demonstrate that workloads are contending for CPU slices rather than processing cooperatively.
• Distinguishing I/O wait from CPU starvation: Standard system load averages aggregate runnable threads and threads sleeping in uninterruptible disk states (D-state). In contrast, runqlen isolates exclusively runnable threads, preventing storage bottlenecks from being misdiagnosed as CPU capacity limits.
• Assessing load balancing efficiency: By breaking down queue statistics per core, administrators can observe whether the kernel scheduler is distributing workloads evenly across NUMA nodes or clustering processes on specific CPU sockets.

Installing BCC Tools on Debian and Ubuntu Systems

To run runqlen, install the standard BCC package and appropriate kernel header files. On modern Debian, Ubuntu, and related distributions, execute the following package management commands with administrative privileges:

sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

On Red Hat Enterprise Linux, Rocky Linux, or Fedora, use the following equivalent commands:

sudo dnf install bcc-tools kernel-devel-$(uname -r)

Once installed, BCC binaries are typically placed inside /usr/sbin on enterprise distributions or suffixed as runqlen-bpfcc under /sbin on Debian and Ubuntu installations.

Executing runqlen to Capture Run Queue Histograms

Running runqlen without command arguments collects system-wide run queue samples every millisecond and summarises the overall queue length distribution when interrupted with Ctrl+C:

sudo /usr/sbin/runqlen-bpfcc

The resulting terminal output displays an ASCII log-2 histogram illustrating how frequently the system had 0, 1, 2, 4, or more runnable processes queued simultaneously:

Sampling run queue length... Hit Ctrl-C to end.
^C
     runqlen       : count     distribution
        0          : 84920    |****************************************|
        1          : 1204     |                                        |
        2          : 84       |                                        |
        4 -> 7     : 12       |                                        |

In this baseline output, the vast majority of samples recorded an empty run queue (value 0), meaning that whenever a task became runnable, an idle CPU core was instantly available to process instructions. Occasional counts in bins 1 through 7 reflect momentary bursts of thread execution that were resolved without persistent queuing backlogs.

Advanced Profiling: Intervals and Per-CPU Granularity

For detailed server diagnostics, runqlen supports interval parameters and hardware topology flags:

1. Outputting Interval Summaries

To monitor queue progression continuously without manual interrupts, specify an interval period in seconds alongside an optional execution count:

sudo /usr/sbin/runqlen-bpfcc 5 3

This command outputs a refreshed histogram every 5 seconds for 3 iterations before exiting, allowing you to observe run queue expansion during scheduled benchmark tests.

2. Decomposing Queue Lengths per CPU Core

When running multi-threaded network engines or containerised microservices, overall system averages can conceal uneven thread distribution. Pass the -C flag to display distinct histograms for each individual processor core:

sudo /usr/sbin/runqlen-bpfcc -C 5 1

If core cpu#0 consistently exhibits queue depths exceeding 8 runnable threads while cores cpu#1 through cpu#7 remain empty, CPU affinity masks (taskset) or hardware IRQ routing routines are improperly confining work to a single physical core.

3. Monitoring Run Queue Saturation Inside Containers

To inspect whether a virtualised workload or container cgroup is starving of CPU shares, pair runqlen with host-level perf tracing to correlate queue depth spikes directly against CFS cgroup quota throttling events.