How to Trace SoftIRQ Execution Time and Event Frequency in Linux Using softirqs

In high-performance Linux computing, enterprise networking, and real-time audio/telecom processing, system responsiveness depends heavily on efficient interrupt handling. When hardware devices—such as Network Interface Cards (NICs) receiving gigabit packet streams or NVMe storage controllers processing I/O requests—trigger hardware interrupts (HardIRQs), the operating system kernel cannot afford to process all associated data immediately inside the top-half interrupt handler. Instead, the Linux kernel defers intensive computation to software interrupts (SoftIRQs), executed asynchronously across dedicated kernel threads (ksoftirqd/x). However, when high-frequency network packets or storage events monopolise SoftIRQ processing, user-space applications experience extreme thread latency, high CPU `%soft` utilization, and jitter. Using the eBPF-powered softirqs utility from the BCC toolkit, systems performance engineers can trace SoftIRQ execution latency and event counts in real time.

Understanding Linux Software Interrupts and Top-Half/Bottom-Half Architecture

The Linux interrupt model is split into two halves:

• Top Half (HardIRQ): The primary interrupt handler executed directly in response to hardware signals. It performs minimum essential operations (such as acknowledging hardware registers) while masking other interrupts, and enqueues the remainder of the work into a SoftIRQ before returning immediately.
• Bottom Half (SoftIRQ): Deferred software execution vectors operating with hardware interrupts re-enabled. There are ten standardised kernel SoftIRQ vectors: HI, TIMER, NET_TX, NET_RX, BLOCK, IRQ_POLL, TASKLET, SCHED, HRTIMER, and RCU.

When SoftIRQ processing takes excessive time per invocation, or when event frequency spikes into millions per second, the kernel cannot service user-space threads. The softirqs tool instruments the kernel tracepoints irq:softirq_entry and irq:softirq_exit using dynamic eBPF probes to capture execution timing and occurrence rates.

Installing the BCC Tracing Toolchain

The softirqs utility is bundled with the BPF Compiler Collection (BCC). Install the necessary eBPF runtime packages and matching kernel headers.

On Debian and Ubuntu distributions, execute:

sudo apt update
sudo apt install -y bpfcc-tools linux-headers-$(uname -r)

On RHEL, Rocky Linux, or Fedora systems, install using dnf:

sudo dnf install -y bcc-tools kernel-devel-$(uname -r)

Verify that the binary is available in your shell search path (often named softirqs-bpfcc on Ubuntu):

which softirqs || which softirqs-bpfcc || ls -l /usr/share/bcc/tools/softirqs

Tracing Cumulative SoftIRQ Service Time

To measure total execution time spent inside each software interrupt vector, execute softirqs with root privileges. By default, running the tool without arguments reports cumulative time in microseconds upon termination via Ctrl + C or after a specified interval:

sudo /usr/sbin/softirqs-bpfcc 5 1

This command records SoftIRQ activity for 5 seconds and displays 1 summary interval:

Tracing soft irq event time... Hit Ctrl-C to end.

SOFTIRQ          TOTAL_usecs
block                     82
tasklet                  145
timer                   1820
sched                   4920
rcu                     8104
net_rx                 38420

In the sample output above, net_rx (network packet reception) consumed over 38 milliseconds of processing time during the 5-second sampling window. On high-throughput web proxies or packet gateways, elevated net_rx time highlights incoming network saturation, indicating a need for Receive Side Scaling (RSS) or NIC hardware ring buffer tuning.

Tracing SoftIRQ Event Counts and Frequencies

In addition to elapsed execution duration, identifying the absolute frequency of interrupt events is crucial for identifying packet storms, timer misconfigurations, or scheduling churn. Use the -c option to report event invocation counts instead of time:

sudo /usr/sbin/softirqs-bpfcc -c 5 1

The output reports total event counts per vector:

SOFTIRQ                TOTAL_count
block                          112
tasklet                        340
timer                         2450
sched                        12104
rcu                          18920
net_rx                       89450

Generating Granular Latency Distribution Histograms

Cumulative totals can obscure bimodal distributions where 99% of interrupts execute in 1 microsecond, but occasional 10-millisecond lockups cause micro-stalls. Use the -d option to output individual power-of-2 latency distribution histograms for each SoftIRQ vector:

sudo /usr/sbin/softirqs-bpfcc -d 10 1

The resulting histograms detail latency distribution across microsecond buckets:

softirq = net_rx
     usecs               : count     distribution
         0 -> 1          : 74210    |****************************************|
         2 -> 3          : 12040    |******                                  |
         4 -> 7          : 2450     |*                                       |
         8 -> 15         : 320      |                                        |
        16 -> 31         : 45       |                                        |
        32 -> 63         : 4        |                                        |

If you observe latency values extending beyond 1,000 microseconds (1 millisecond), kernel interrupt processing is causing severe scheduling delays on that CPU core.

Adding Timestamps for Periodic Monitoring

For ongoing observability in production environments, pass the -T option alongside a sampling interval in seconds to prepend standard timestamps to each printed report:

sudo /usr/sbin/softirqs-bpfcc -T 60

This allows systems administrators to correlate interrupt surges with external application metrics, API latency spikes, or sudden network DDoS incidents recorded in centralized observability pipelines.