In low-latency Linux computing, high-frequency financial trading, and real-time audio and industrial control systems, maintaining deterministic scheduling requires that hardware interrupt service routines execute as rapidly as possible. When a physical peripheral—such as an NVMe storage controller, a 100GbE network interface, or a PCIe bus-mastering accelerator—signals the CPU via an interrupt line, the processor immediately halts user-space execution and enters the kernel’s top-half interrupt handler (HardIRQ). While operating inside a HardIRQ handler, local CPU hardware interrupts are masked. If a poorly optimized device driver or failing hardware component spends excessive time servicing a HardIRQ, other pending hardware interrupts are delayed, real-time audio drops frames, and user-space threads experience severe micro-stalls. By using the eBPF-powered hardirqs utility from the BCC toolchain, systems performance engineers can record exact hardware interrupt servicing latency and event frequencies in real time.
Understanding Linux Hardware Interrupts and Kernel Top Halves
Hardware interrupts (HardIRQs) are asynchronous signals generated by physical peripheral chips to alert the CPU that an I/O operation has completed or requires immediate attention. Because HardIRQs run with interrupts disabled on the executing CPU core, modern operating system architectures adhere to strict design constraints:
- Minimal Execution Footprint: The HardIRQ handler must perform only the minimum necessary operations—such as reading hardware status registers, acknowledging the interrupt controller (APIC), clearing physical line flags, and enqueuing deferred work to SoftIRQs or threaded IRQ handlers.
- Latency Sensitivity: A well-behaved HardIRQ should typically complete within a few microseconds (often under 5 microseconds). If a HardIRQ handler takes tens or hundreds of microseconds, it blocks scheduler ticks and introduces jitter into user applications.
The hardirqs tool attaches dynamic eBPF probes to the Linux kernel tracepoints irq:irq_handler_entry and irq:irq_handler_exit. It calculates the elapsed time between entry and exit timestamps, attributing execution latency directly to the specific hardware interrupt line and registered driver name.
Installing the BCC eBPF Tracing Toolchain
The hardirqs utility is distributed as part of the standard BPF Compiler Collection (BCC). It requires active eBPF kernel support and kernel development headers.
On Debian and Ubuntu systems, install BCC tools and matching headers using apt:
sudo apt update
sudo apt install -y bpfcc-tools linux-headers-$(uname -r)On enterprise distributions such as RHEL, Rocky Linux, or Fedora, install via dnf:
sudo dnf install -y bcc-tools kernel-devel-$(uname -r)Confirm the executable path on your system (often suffixed with -bpfcc on Debian/Ubuntu):
which hardirqs || which hardirqs-bpfcc || ls -l /usr/share/bcc/tools/hardirqsTracing Cumulative HardIRQ Service Times
To measure the cumulative time consumed by each hardware interrupt line across all CPU cores, execute hardirqs with root privileges. By default, running the tool reports cumulative elapsed time in microseconds upon termination via Ctrl + C or after a specified interval:
sudo /usr/sbin/hardirqs-bpfcc 5 1This command records hardware interrupt activity for 5 seconds and outputs 1 summary interval:
Tracing hard irq event time... Hit Ctrl-C to end.
HARDIRQ TOTAL_usecs
nvme0q1 142
eth0-TxRx-0 1205
eth0-TxRx-1 1840
i915 3412
xhci_hcd 8950The summary table details the registered interrupt handler name and total elapsed processing time. In the example above, the USB 3 host controller (xhci_hcd) and the GPU driver (i915) consumed the largest shares of interrupt service time.
Tracing Hardware Interrupt Occurrence Frequencies
In addition to latency duration, monitoring the total event count generated by each peripheral helps identify interrupt storms caused by hardware faults or misconfigured line polling. Use the -c option to report event frequency counts instead of elapsed time:
sudo /usr/sbin/hardirqs-bpfcc -c 5 1The resulting table displays total occurrences per interrupt line:
HARDIRQ TOTAL_count
nvme0q1 412
i915 1820
xhci_hcd 3450
eth0-TxRx-0 14820
eth0-TxRx-1 16240Generating Latency Distribution Histograms
A low cumulative service time can conceal occasional severe latency spikes where a single interrupt routine stalls the CPU for milliseconds. To expose outlier latency, use the -d option to render power-of-2 distribution histograms for each active interrupt handler:
sudo /usr/sbin/hardirqs-bpfcc -d 10 1The command outputs detailed histograms formatted in microseconds:
hardirq = eth0-TxRx-0
usecs : count distribution
0 -> 1 : 12450 |****************************************|
2 -> 3 : 3410 |********** |
4 -> 7 : 420 |* |
8 -> 15 : 32 | |
16 -> 31 : 4 | |
32 -> 63 : 0 | |If you observe counts in the buckets above 64 microseconds—or extending into milliseconds—the target driver is performing excessive computation inside its top half, warranting threaded IRQ conversion or driver optimization.
Reporting Latency in Milliseconds for Overloaded Systems
On servers experiencing extreme hardware degradation or virtualization interrupt injection stalls, microsecond scales can overflow visual formatting. Pass the -m flag to convert histogram bucket ranges directly to milliseconds:
sudo /usr/sbin/hardirqs-bpfcc -d -m 10 1Adding Timestamps for Periodic Health Audits
For ongoing telemetry collection and automated anomaly detection, use the -T option alongside an interval argument in seconds to prepend standard timestamps to each printed report:
sudo /usr/sbin/hardirqs-bpfcc -T 60This allows systems performance engineers to correlate hardware interrupt spikes with network traffic surges, NVMe storage benchmarks, or application tail latency events recorded in centralized monitoring platforms.