How to Trace Process Wakeups and Latency in Linux Using wakeuptime

In operating systems performance analysis, identifying why application threads experience intermittent latency spikes requires measuring scheduler latency. While runqlat measures the duration a thread spends waiting on a CPU run queue after becoming runnable, it cannot reveal the preceding event that awakened the sleeping thread. When applications exhibit tail latency spikes in web APIs, databases, or high-frequency trading platforms, understanding which processes or kernel interrupts trigger task wakeups is crucial.

A thread sleeping on an I/O wait, timer, or futex lock does not wake itself; an external event—such as a storage controller interrupt, an inbound network packet, or another thread releasing a mutex—invokes the kernel’s scheduler wakeup mechanism. The BPF Compiler Collection (BCC) provides wakeuptime, an eBPF utility designed specifically to record task wakeups. It captures the target thread, the awakening process, the duration the target was asleep, and the full kernel stack trace that issued the wakeup. This guide explains how to trace process wakeup latency and relationships in Linux using wakeuptime.

How wakeuptime Operates via eBPF

The wakeuptime utility leverages extended Berkeley Packet Filters (eBPF) to hook into the Linux kernel’s core scheduler wakeup functions:

• ttwu_do_wakeup (Try To Wake Up): The kernel internal function executed when a task is moved from a wait queue into a runnable state.
• finish_task_switch: Captures when tasks transition into sleep states to calculate total sleep duration.

Whenever a waking event occurs, wakeuptime records:

1. The waker process command name and Process ID (the thread issuing the wakeup).
2. The target sleeping process command name and PID (the thread being awakened).
3. The kernel call stack trace of the waker process showing why it woke the target.
4. The total elapsed sleep duration in microseconds ($\mu\text{s}$).

By aggregating sleep durations into an in-kernel BPF hash map keyed by the unique combination of waker stack, waker process, and target task, wakeuptime captures wakeup relationships with negligible system overhead.

Prerequisites and Tool Installation

The utility requires Linux kernel 4.9 or newer with BPF enabled, along with elevated administrative permissions.

Installing on Debian and Ubuntu

1. Update the local APT repository cache:

sudo apt update

2. Install the BCC tools package and Linux kernel headers:

sudo apt install bpfcc-tools linux-headers-$(uname -r)

The executable is available as wakeuptime-bpfcc under /usr/sbin/.

Installing on RHEL, Fedora, and Rocky Linux

Install the package using dnf:

sudo dnf install bcc-tools kernel-devel-$(uname -r)

The binary is located at /usr/share/bcc/tools/wakeuptime.

Tracing Wakeup Relationships System-Wide

To begin tracing wakeup events system-wide for a fixed duration of 10 seconds, run the command with root privileges:

sudo wakeuptime-bpfcc 10

When the sampling window elapses, wakeuptime prints the unique waker call stacks, the interacting process names, and the cumulative sleep durations:

    try_to_wake_up
    default_wake_function
    autoremove_wake_function
    wake_up_process
    ep_poll_callback
    __wake_up_common
    __wake_up_sync_key
    sock_def_readable
    tcp_rcv_established
    tcp_v4_do_rcv
    tcp_v4_rcv
    ip_protocol_deliver_rcu
    ip_local_deliver
    ip_sublist_rcv
    -                ksoftirqd/0 (PID: 12)
    --               node (PID: 3810)
        1248019 microseconds

Interpreting the Output Hierarchy

• Stack Trace: Displays the kernel functions executed by the waker. In the example above, network softirq processing (ksoftirqd) handled incoming TCP packets, invoked sock_def_readable, and triggered an epoll wake callback (ep_poll_callback).
• Waker Process: ksoftirqd/0 (PID: 12)—the kernel interrupt daemon that initiated the wakeup.
• Target Process: node (PID: 3810)—the application process that was sleeping waiting for network data.
• Duration: 1248019 microseconds—the cumulative duration the target process spent asleep before being awakened by this specific stack.

Advanced Filtering and Target Isolation

On high-traffic systems, capturing all scheduler wakeups produces voluminous output. Use command-line arguments to isolate specific applications and investigate targeted latency issues.

Filtering by Target Process ID

To determine what kernel events or external processes are waking a specific application thread, pass the target Process ID to -p:

sudo wakeuptime-bpfcc -p 3810 5

Filtering Minimum and Maximum Sleep Durations

To isolate transient lock contention or fast wakeup loops, use the -m (min duration in microseconds) and -M (max duration in microseconds) parameters:

sudo wakeuptime-bpfcc -p 3810 -m 100 -M 50000 10

This filters out long idle periods and focuses exclusively on sleep intervals between 0.1ms and 50ms, highlighting rapid lock handoffs and IPC synchronization delays.

User-Space Stack Resolution

To inspect user-space function names of the waker thread when one application process wakes another (e.g. via pipes, UNIX domain sockets, or futexes), append the -u flag:

sudo wakeuptime-bpfcc -u 10

Generating Wakeup Flame Graphs

Complex wakeup call chains across multi-threaded applications are easiest to analyze visually using flame graphs:

1. Capture folded wakeup stacks using the -f flag:

sudo wakeuptime-bpfcc -df 10 > out.wakeuptime-stacks

2. Convert the output into an interactive SVG flame graph:

flamegraph.pl --color=chain --title="Wakeup Time Flame Graph" out.wakeuptime-stacks > wakeuptime.svg

In a wakeup flame graph, horizontal span represents cumulative sleep duration, making it straightforward to identify which kernel subsystems and background worker threads are responsible for resuming blocked workloads.

Performance and Operational Safeguards

• Low Overhead: Like other BCC tools, wakeuptime performs in-kernel aggregation and dumps top records upon exit, keeping CPU consumption negligible during profiling.
• Scheduler Contention: On systems with extremely high context switch rates (e.g. hundreds of thousands per second), prefer targeted profiling with -p to minimize scheduler tracepoint overhead.
• Paired Analysis: Use wakeuptime alongside offcputime. While offcputime shows why a thread went to sleep, wakeuptime shows what woke it back up, providing a 360-degree view of blocking latency.