How to Trace Off-CPU Time and Blocking Latency in Linux Using offcputime

When profiling high-latency applications and diagnosing system stalls in Linux, performance analysts traditionally focus on CPU profiling tools like top or perf record. These utilities sample the system while threads are on-CPU actively burning cycles. However, the majority of application latency issues—such as slow database query responses, unresponsive web endpoints, and microservice deadlocks—occur when threads are completely off-CPU, blocked waiting on locks, filesystem I/O, network sockets, or paging events.

Because blocked threads do not consume CPU cycles, conventional on-CPU sampling profilers cannot detect them. To resolve these performance mysteries, engineers must measure off-CPU time—the duration tasks spend sleeping in wait queues—alongside the exact kernel and user-space call stacks that led to the blocking state. The BPF Compiler Collection (BCC) provides offcputime, an eBPF tracing utility designed specifically to record blocking latency and kernel stack traces. This guide shows how to trace off-CPU blocking latency in Linux using offcputime.

How offcputime Operates via eBPF

The offcputime utility hooks directly into the Linux kernel’s process scheduler using the finish_task_switch or sched_switch kernel tracepoints. Its operational sequence proceeds as follows:

1. When an active thread is descheduled (swapped out of the CPU), offcputime records a high-resolution timestamp and captures the thread’s current user-space and kernel stack trace.
2. When that same thread is eventually scheduled back onto a CPU core, offcputime calculates the elapsed elapsed time ($\Delta t$) spent blocked off-CPU.
3. Instead of streaming individual switch events to user space (which would introduce immense scheduler overhead), offcputime aggregates the cumulative sleep durations directly into an in-kernel BPF hash map keyed by the unique call stack.
4. When execution ends, it summarises the top blocking paths sorted by cumulative off-CPU time.

Prerequisites and Tool Installation

The utility requires Linux kernel 4.9 or newer with BPF enabled, along with elevated administrative permissions.

Installing on Debian and Ubuntu

1. Update the local APT repository cache:

sudo apt update

2. Install the BCC tools package and Linux kernel headers:

sudo apt install bpfcc-tools linux-headers-$(uname -r)

The executable is available as offcputime-bpfcc under /usr/sbin/.

Installing on RHEL, Fedora, and Rocky Linux

Install the package using dnf:

sudo dnf install bcc-tools kernel-devel-$(uname -r)

The executable resides in /usr/share/bcc/tools/offcputime.

Tracing Off-CPU Time Across the Entire System

To begin tracing off-CPU sleep intervals system-wide for 10 seconds, execute the command with root privileges and specify the duration in seconds:

sudo offcputime-bpfcc 10

When the 10-second sampling window completes, offcputime prints unique call stacks accompanied by the cumulative microseconds spent waiting:

    finish_task_switch
    schedule
    schedule_timeout
    wait_woken
    sk_wait_data
    tcp_recvmsg
    inet_recvmsg
    sock_read_iter
    new_sync_read
    vfs_read
    ksys_read
    __x64_sys_read
    do_syscall_64
    entry_SYSCALL_64_after_hwframe
    __libc_read
    -                node (PID: 3810)
        4521098 microseconds

Interpreting the Stack Trace

The stack trace above reads from the bottom (user space) to the top (kernel wait queue):

• The Node.js process (PID 3810) invoked a read() system call via libc.
• The kernel transitioned the request through vfs_read into the TCP networking layer (tcp_recvmsg).
• Because no network packets were ready in the socket receive buffer, the kernel called sk_wait_data and put the task to sleep via schedule().
• The thread spent over 4.52 cumulative seconds blocked waiting for inbound network traffic across the 10-second window.

Advanced Filtering and Target Isolation

System-wide off-CPU tracing captures hundreds of background kernel worker threads. Use command-line arguments to isolate specific applications and filter out intentional long sleeps.

Filtering by Process ID

To inspect why a specific database or backend daemon is stalling, restrict tracing to its specific Process ID using the -p parameter:

sudo offcputime-bpfcc -p 1420 5

Filtering Minimum and Maximum Block Durations

Idle daemon threads often sleep for long periods using epoll_wait() or nanosleep(), dominating the output with intentional idle waits. Use the -m (min duration in microseconds) and -M (max duration in microseconds) flags to isolate specific latency windows:

sudo offcputime-bpfcc -p 1420 -m 1000 -M 100000 10

This command captures only blocking events lasting between 1ms (1,000 µs) and 100ms (100,000 µs), isolating transient lock contention and disk I/O stalls while ignoring idle background sleep loops.

User-Space Stacks Only

If you only need to see the application code functions responsible for blocking calls without kernel internal frames, append the -u flag:

sudo offcputime-bpfcc -u -p 1420 10

Generating Off-CPU Flame Graphs

Visualising complex nested stack traces across high-throughput systems is vastly simplified using Brendan Gregg’s FlameGraph toolkit:

1. Capture raw stack outputs into a trace file with folded stacks using the -f flag:

sudo offcputime-bpfcc -df 10 > out.offcpu-stacks

2. Render the folded stacks into an interactive SVG flame graph:

flamegraph.pl --color=io --title="Off-CPU Time Flame Graph" out.offcpu-stacks > offcpu.svg

In an Off-CPU flame graph, horizontal bar width represents the total time spent blocked, allowing you to identify lock contention and synchronous storage bottlenecks immediately.

Performance and Operational Safeguards

• Debug Symbols Required: To resolve meaningful function names in user-space stack traces, the target application binary must be compiled with frame pointers (-fno-omit-frame-pointer) and retain debugging symbols.
• High Context-Switch Overhead: On systems experiencing millions of context switches per second, running system-wide off-CPU tracing can introduce noticeable CPU overhead. Always prefer scoping tracing to a specific PID (-p) or a brief 5-to-10 second sampling window.