When profiling high-concurrency Linux applications, developers often focus exclusively on on-CPU profiling—measuring which functions consume CPU cycles while actively executing on cores. However, many severe application performance degradation issues and tail-latency spikes occur while threads are off-CPU: blocked waiting for disk I/O, suspended by mutex lock contention, stalling on network socket reads, or paused by kernel memory reclamation. The BCC eBPF tool offcputime captures and quantifies the exact stack traces responsible for off-CPU duration.
Understanding the Mechanics of Off-CPU Profiling
Standard CPU profilers (such as perf record or sampling profilers) sample instruction pointers at fixed timer intervals. If a thread is blocked in an uninterruptible sleep state (TASK_UNINTERRUPTIBLE or TASK_INTERRUPTIBLE), the hardware timer does not interrupt it, rendering the stall invisible to on-CPU flame graphs.
offcputime resolves this by attaching eBPF probes directly to the Linux scheduler tracepoint sched:sched_switch:
- When a thread is scheduled off a CPU core,
offcputimerecords a high-resolution kernel timestamp and saves the process ID and kernel/user call stacks. - When the thread is subsequently scheduled back onto an available core, the tool computes the elapsed off-CPU time.
- It aggregates identical call stacks into an in-kernel BPF hash map, keeping runtime tracing overhead exceptionally low even on high-load production servers.
Installing the BCC Tools Suite
To use offcputime, verify that the BPF Compiler Collection package and kernel development headers are installed:
# Ubuntu / Debian
sudo apt-get update
sudo apt-get install -y bpfcc-tools linux-headers-$(uname -r)
# RHEL 9 / Rocky Linux 9 / AlmaLinux
sudo dnf install -y bcc-tools kernel-devel-$(uname -r)
# Verify executable binary path
which offcputime || which /usr/share/bcc/tools/offcputime
Tracing Off-CPU Time Across Specific Processes or System-Wide
Launch offcputime specifying the target duration (in seconds) and optional filtering criteria:
# Trace off-CPU time for PID 18420 for 10 seconds
sudo /usr/share/bcc/tools/offcputime -p 18420 10
# Capture only blocking events that stalled for at least 1,000 microseconds (1 ms)
sudo /usr/share/bcc/tools/offcputime -p 18420 -m 1000 10
# Trace only user-space call stacks (excluding kernel internals)
sudo /usr/share/bcc/tools/offcputime -u -p 18420 5
Interpreting Stack Traces and Blocking Latency
Upon completing the tracing window, offcputime prints aggregated unique call stacks ordered from least to most accumulated blocking time, terminating with the total off-CPU duration in microseconds:
futex_wait_queue_me
futex_wait
do_futex
__x64_sys_futex
do_syscall_64
entry_SYSCALL_64_after_hwframe
pthread_cond_wait@@GLIBC_2.3.2
ThreadPoolExecutor::Worker::run()
- mysqld (18420)
4215890
io_schedule
wait_on_page_bit_common
__filemap_get_folio
xfs_file_buffered_read
xfs_file_read_iter
new_sync_read
vfs_read
ksys_read
do_syscall_64
- mysqld (18420)
8912440
In this output:
- The first stack reveals that worker threads spent 4.2 seconds blocked in
pthread_cond_wait(futex sleep), which is normal idle thread-pool behaviour. - The second stack reveals that the process spent 8.9 seconds blocked in
io_schedulewaiting for page-cache reads from the XFS filesystem, directly pointing to storage controller or page eviction stalls.
Generating Off-CPU Flame Graphs for Forensic Analysis
Raw text stacks can be difficult to interpret across large multithreaded systems. Combine offcputime with Brendan Gregg’s FlameGraph generator to produce interactive visualisations:
# Record raw folded stacks
sudo /usr/share/bcc/tools/offcputime -df 10 > out.offcpu-folded
# Clone FlameGraph repository and render SVG
git clone https://github.com/brendangregg/FlameGraph.git
./FlameGraph/flamegraph.pl --color=io --title="Off-CPU Time Flame Graph" out.offcpu-folded > offcpu.svg
In the resulting blue-hued “cold” flame graph, the widest towers represent the exact functions and system calls causing your application threads to spend the greatest aggregate duration waiting off-CPU.
By integrating offcputime into your performance debugging arsenal, systems engineers can pinpoint hidden lock contention and storage stalls that traditional CPU utilization tools completely miss.