How to Trace Kernel Workqueues and IRQ Latency in Linux Using perf kwork

In high-throughput Linux servers, real-time audio/video processing systems, and low-latency networking environments, unexpected latency spikes often stem from asynchronous kernel execution. Hardware Interrupt Requests (IRQs), Software Interrupts (softirqs), and deferred kernel workqueues executed by kworker threads run at high CPU priority, temporarily preempting user-space applications and real-time threads.

Historically, isolating whether a latency spike was caused by a slow network driver interrupt, an unoptimized storage completion handler, or an overburdened kernel workqueue required stitching together custom tracepoints using Ftrace or eBPF. Modern Linux kernels (Linux 5.18 and later) streamline this diagnostic workflow through the perf kwork subsystem. Built directly into the Linux perf toolkit, perf kwork records, measures, and reports the exact execution time, scheduling latency, and CPU overhead of all kernel interrupt and workqueue tasks.

Understanding perf kwork Architectural Concepts

The perf kwork tool tracks three distinct classes of deferred and interrupt-driven kernel activity:

• irq (Hard IRQ): Hardware interrupt handlers triggered by physical peripheral devices (such as NICs, NVMe storage controllers, or system timers).
• softirq (Soft IRQ): Bottom-half interrupt handlers that process deferred work (such as network packet processing via NET_RX or timer ticks via TIMER_SOFTIRQ).
• workqueue (kworker): Asynchronous kernel worker threads executing deferred functions scheduled by kernel subsystems and device drivers.

For each event, perf kwork measures both scheduling delay (the latency between when an interrupt or work item was raised and when execution actually began) and runtime duration (how long the handler occupied the CPU).

Verifying Kernel and Perf Support

The perf kwork subcommand requires kernel support for tracepoints in the interrupt and workqueue subsystems. Verify that perf is installed on your Linux system and supports the kwork command:

perf kwork --help

Ensure that tracefs is mounted and kernel profiling permissions are configured. For complete kernel event tracing across all CPU cores, run commands with administrative privileges or configure perf_event_paranoid:

sudo sysctl -w kernel.perf_event_paranoid=1

Recording Kernel Work and IRQ Events with perf kwork record

To record kernel interrupt and workqueue events system-wide, use perf kwork record. You can target all subsystems or restrict collection to specific event classes using -b (events):

# Record all kernel workqueue, irq, and softirq events for 5 seconds
sudo perf kwork record -- sleep 5

# Record only hardware IRQ and softirq events
sudo perf kwork record -b irq,softirq -- sleep 5

# Record only kernel workqueue threads
sudo perf kwork record -b workqueue -- sleep 5

During execution, perf captures timestamped tracepoint events and saves them to a local binary trace file named perf.data.

Analysing Overall Kernel Overhead with perf kwork report

Once recording completes, generate an overview report detailing the total time consumed by each kernel worker and interrupt handler:

sudo perf kwork report

The output presents a formatted table sorted by CPU time:

Kwork Name              | Ins | Total Time |   Avg Time |   Max Time | Max Time At
----------------------------------------------------------------------------------
[s] NET_RX              | 840 |   18.42 ms |   21.92 us |  245.10 us | 12450.812301
[k] e1000e_poll_task    | 412 |    8.14 ms |   19.75 us |  112.45 us | 12450.814210
[i] eth0 (irq 16)       | 840 |    1.82 ms |    2.16 us |   15.20 us | 12450.810145
[s] TIMER               | 250 |    0.95 ms |    3.80 us |   28.10 us | 12450.815901

The type column indicates whether the task is a softirq ([s]), kernel workqueue ([k]), or hard IRQ ([i]). Reviewing Max Time identifies handlers that blocked CPU cores for extended durations.

Diagnosing Latency and Scheduling Delay with perf kwork latency

To identify responsiveness bottlenecks where high-priority work items waited too long before obtaining CPU time, run the latency report:

sudo perf kwork latency

The latency report highlights the scheduling delay across tasks:

Kwork Name              | Ins | Total Delay |  Avg Delay |  Max Delay | Max Delay At
----------------------------------------------------------------------------------
[k] flush_to_ldisc      |  45 |    1.42 ms  |   31.55 us |  185.40 us | 12450.820112
[s] NET_RX              | 840 |    0.85 ms  |    1.01 us |   42.10 us | 12450.811400
[i] eth0 (irq 16)       | 840 |    0.12 ms  |    0.14 us |    2.50 us | 12450.810050

A high Max Delay indicates that CPU cores were locked by other high-priority interrupts or disabled preemption regions, preventing the worker from running promptly.

Identifying Real-Time CPU Contention with perf kwork top

To inspect tasks that consumed the largest proportion of total recording time during the sampling window:

sudo perf kwork top

The top view aggregates instances, total CPU runtime, and runtime percentage, providing an immediate snapshot of whether networking, storage, or timer interrupts dominate kernel processing time.

Filtering and Customising Trace Outputs

When debugging specific drivers (such as a wireless NIC or RAID controller), filter output directly by name or CPU core index:

# Filter by specific device or handler name
sudo perf kwork report --name "eth0"

# Restrict report to CPU core 0
sudo perf kwork report --cpu 0

By using perf kwork to audit kernel workqueues and interrupt latency, Linux systems administrators and kernel developers can eliminate invisible scheduling delays, optimize driver interrupt balancing across CPU cores, and ensure consistent real-time application throughput.