How to Measure Task On-CPU Execution Time in Linux Using cpudist

When profiling high-throughput multi-threaded applications, database engines, or real-time microservices on Linux, traditional performance monitoring tools such as top, htop, or vmstat report aggregated CPU percentage utilization across sampling intervals. However, CPU utilization percentages obscure the micro-architectural behaviour of individual threads: they cannot reveal whether a process is running in long, unbroken bursts of compute work or experiencing thousands of rapid, fragmented context switches caused by lock contention or voluntary I/O yields.

The BPF Compiler Collection (BCC) provides cpudist, an extended Berkeley Packet Filter (eBPF) diagnostic tool designed to measure on-CPU task execution duration. By hooking directly into the Linux Completely Fair Scheduler (CFS) context-switch tracepoints, cpudist captures the exact duration that threads remain actively scheduled on physical CPU cores before yielding or getting preempted, outputting high-resolution power-of-2 logarithmic histograms.

Understanding On-CPU Time vs Off-CPU Time in Scheduler Profiling

Operating system thread execution alternates between two fundamental operational states:

• On-CPU Time: The thread is actively executing machine instructions on a hardware CPU core. cpudist measures the elapsed time from when a thread is scheduled on a CPU (context switch in) until it yields the CPU (context switch out).
• Off-CPU Time: The thread is blocked waiting for events — such as disk I/O, network socket data, mutex locks, page faults, or timer sleeps (profiled separately by tools like offcputime).

By analysing on-CPU distribution histograms, performance engineers can distinguish between two critical performance profiles:

• Compute-Bound Execution (Long Bursts): Threads execute for hundreds of milliseconds uninterrupted, characteristic of mathematical rendering, batch processing, or unoptimised cryptographic hashing.
• Lock Contention / Involuntary Preemption (Short Slices): Threads execute for only 1 to 5 microseconds before yielding, revealing excessive spinlock contention, heavy context-switching overhead, or scheduler thrashing.

Installing BCC Tools on Modern Linux Distributions

To run cpudist, install the BCC tools package corresponding to your Linux distribution.

On Ubuntu and Debian systems:

sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

(On Ubuntu, BCC binaries are installed with a -bpfcc suffix, e.g. cpudist-bpfcc.)

On Red Hat Enterprise Linux, Rocky Linux, or Fedora:

sudo dnf install bcc-tools kernel-devel-$(uname -r)

On Arch Linux:

sudo pacman -S bcc bcc-tools linux-headers

Running cpudist to Profile On-CPU Distributions

To record on-CPU durations across the entire system, execute cpudist with root privileges. By default, running without arguments tracks durations in microsecond buckets until interrupted (press Ctrl + C):

sudo cpudist-bpfcc

(On RHEL/CentOS, execute sudo /usr/share/bcc/tools/cpudist.)

A typical on-CPU histogram displays elapsed time ranges alongside visual ASCII distribution bars:

Tracing on-CPU time... Hit Ctrl-C to end.
^C
     usecs               : count     distribution
         0 -> 1          : 1240      |**                                      |
         2 -> 3          : 4510      |*********                               |
         4 -> 7          : 18450     |****************************************|
         8 -> 15         : 9210      |********************                    |
        16 -> 31         : 3120      |******                                  |
        32 -> 63         : 840       |*                                       |
        64 -> 127        : 120       |                                        |
       128 -> 255        : 45        |                                        |
       256 -> 511        : 12        |                                        |

The distribution peaks sharply between 4 and 7 microseconds, indicating that tasks are executing tiny chunks of instructions before relinquishing processor cores to other tasks or blocking on system calls.

Targeting Specific Applications and Processes

System-wide tracing aggregates all background daemons, kernel worker threads, and desktop processes. To isolate a specific mission-critical application (such as PostgreSQL or an internal Java worker), filter by Process ID (PID) using the -p switch:

# Find the PID of your target service
pgrep -f redis-server

# Trace only Redis execution slices
sudo cpudist-bpfcc -p 4812

To measure on-CPU time on a per-thread level (rather than aggregating all threads into the parent process), pass the -T flag:

sudo cpudist-bpfcc -p 4812 -T

This generates independent histograms for each OS thread (TID), allowing you to identify whether a single worker thread is hogging CPU cores while sibling threads stall.

Measuring in Milliseconds for Batch and High-Compute Workloads

For batch data analytics, video encoding, or AI inference models where threads run for sustained intervals, microsecond granularity is too fine. Pass the -m flag to switch the histogram buckets to milliseconds:

sudo cpudist-bpfcc -m -p $(pgrep ffmpeg)

Output in millisecond buckets:

     msecs               : count     distribution
         0 -> 1          : 85        |*                                       |
         2 -> 3          : 120       |**                                      |
         4 -> 7          : 410       |********                                |
         8 -> 15         : 1890      |****************************************|
        16 -> 31         : 1420      |******************************          |
        32 -> 63         : 510       |**********                              |

Here, the peak between 8 and 15 milliseconds aligns directly with standard Linux scheduler timeslices (sysctl kernel.sched_latency_ns), confirming uninterrupted compute-bound execution.

Sampling over Timed Intervals

To capture periodic snapshots during performance stress tests without manually interrupting the session, specify an interval (in seconds) and count:

# Output a summary every 5 seconds, running 3 times
sudo cpudist-bpfcc 5 3

This prints three consecutive 5-second histograms, allowing you to observe shifts in on-CPU patterns before, during, and after artificial traffic spikes.

Using the bpftrace Implementation

If full Python-based BCC tools cannot be installed in minimal staging environments, modern distributions include a standalone cpudist.bt script powered by bpftrace:

sudo bpftrace /usr/sbin/cpudist.bt

Or profile on-CPU times for a target PID using a direct bpftrace one-liner:

sudo bpftrace -e '
tracepoint:sched:sched_switch {
  if (args->prev_pid == 4812) {
    @start[args->prev_pid] = nsecs;
  }
  if (args->next_pid == 4812) {
    $last = @start[args->next_pid];
    if ($last) {
      @on_cpu_us = hist((nsecs - $last) / 1000);
      delete(@start[args->next_pid]);
    }
  }
}'

This provides an immediate, low-overhead histogram without installing runtime compilers or Python toolkits.