How to Inspect Hardware Interrupts and Configure IRQ Affinity in Linux Using lsirq and procfs

Modern high-throughput computing platforms—such as 100 GbE network gateways, ultra-low-latency financial trading servers, and real-time audio processors—process hundreds of thousands of hardware interrupts every second. When peripheral devices (such as NVMe controllers, network interface cards, or GPU accelerators) signal the CPU that new data packets or I/O blocks are waiting in memory, the kernel pauses user-space threads to execute interrupt service routines (ISRs). If hardware interrupts from multiple high-traffic peripherals route to the same CPU core (typically Core 0), that core suffers severe cache thrashing, high latency spikes, and packet drops while neighboring cores remain completely idle. Through modern inspection tools such as lsirq alongside procfs virtual files, Linux administrators can audit interrupt distributions and pin specific hardware lines to isolated CPU cores using symmetric multiprocessing (SMP) IRQ affinity.

Understanding Hardware Interrupts and IRQ Line Allocation

In modern x86-64 systems, hardware interrupts operate via Message Signaled Interrupts (MSI / MSI-X) or Advanced Programmable Interrupt Controllers (APIC). Instead of relying on legacy shared physical wire lines, MSI-X allows peripherals to allocate dozens or hundreds of dedicated interrupt vectors directly across PCIe interconnects.

Managing hardware interrupts involves two core kernel abstractions:

• /proc/interrupts: Real-time accounting of interrupt counts received by every CPU core across all registered IRQ lines, including peripheral controllers and architecture-specific interrupts (such as Local APIC timer interrupts, Rescheduling interrupts, and TLB shootdowns).
• /proc/irq/[IRQ]/smp_affinity: A hexadecimal bitmask that dictates which specific CPU cores are eligible to service interrupts originating from that IRQ line.

Inspecting Active Interrupt Lines Using lsirq

While reading /proc/interrupts directly is standard practice, modern Linux distributions include lsirq (part of the util-linux suite), which provides structured, sortable terminal views of interrupt lines.

To inspect active hardware interrupts sorted by the highest interrupt count:

lsirq --sort=total

To view real-time changes with column formatting:

lsirq -c -o IRQ,TOTAL,NAME

The output reports the IRQ number, the hardware device driver (e.g. nvme0q0, mlx5_comp0, xhci_hcd), and total occurrences across CPUs.

To read the raw kernel interrupt matrix across individual CPU cores:

watch -n 1 "cat /proc/interrupts | head -n 25"

Observe column distribution: if CPU0 displays millions of counts while CPU1 through CPU15 report zero, interrupt balancing is unconfigured or misaligned.

Understanding Hexadecimal SMP Affinity Bitmasks

IRQ affinity is expressed as a hexadecimal bitmask where each binary bit corresponds to a logical processor core:

• Core 0: 0001 (Binary: 0000 0001) -> Hex: 1
• Core 1: 0002 (Binary: 0000 0010) -> Hex: 2
• Core 2: 0004 (Binary: 0000 0100) -> Hex: 4
• Core 3: 0008 (Binary: 0000 1000) -> Hex: 8
• Cores 0-3: 000f (Binary: 0000 1111) -> Hex: f
• Core 4: 0010 (Binary: 0001 0000) -> Hex: 10

Alternatively, the kernel provides smp_affinity_list, which accepts comma-separated CPU indices or hyphenated ranges (e.g. 2,4-7), eliminating hexadecimal conversion.

Manually Binding an IRQ Line to a Dedicated CPU Core

Before configuring manual affinity, ensure the automated dynamic interrupt balancer (irqbalance) does not overwrite your custom pinning:

# Check if irqbalance daemon is running
systemctl status irqbalance

# If manual deterministic pinning is required, mask or stop irqbalance
sudo systemctl stop irqbalance

To locate the IRQ numbers associated with your primary network interface (e.g. eth0 or enp3s0):

grep -E "eth0|enp3s0" /proc/interrupts

Suppose your high-speed NIC queue 0 uses IRQ 125. To pin IRQ 125 strictly to CPU Core 2:

# Method 1: Using smp_affinity_list (Recommended)
echo "2" | sudo tee /proc/irq/125/smp_affinity_list

# Method 2: Using hexadecimal bitmask (Core 2 = bit 2 = 0x4)
echo "4" | sudo tee /proc/irq/125/smp_affinity

To verify that the affinity took effect:

cat /proc/irq/125/smp_affinity_list
cat /proc/irq/125/smp_affinity

Distributing Multi-Queue Network Interfaces Across CPU Cores

Modern NICs implement Receive Side Scaling (RSS) with multiple RX/TX queues. To maximize packet processing throughput, each hardware queue should be bound to a unique, non-overlapping physical CPU core.

You can automate queue affinity distribution with a lightweight shell script:

#!/usr/bin/env bash
# Distribute eth0 queues across CPU cores 4 through 11
INTERFACE="eth0"
START_CPU=4

IRQS=$(grep "$INTERFACE" /proc/interrupts | awk '{print $1}' | tr -d ':')
CURRENT_CPU=$START_CPU

for IRQ in $IRQS; do
    echo "Binding IRQ $IRQ to CPU Core $CURRENT_CPU"
    echo "$CURRENT_CPU" | sudo tee /proc/irq/$IRQ/smp_affinity_list > /dev/null
    ((CURRENT_CPU++))
done

echo "Queue affinity distribution completed."

Auditing IRQ Distribution Under Synthetic Traffic

To confirm that incoming packet workloads are evenly distributed without core saturation, run mpstat while generating network or disk load:

mpstat -P ALL 1

Examine the %irq (hard interrupts) and %soft (softirqs/ksoftirqd) columns. You will observe hard interrupt processing occurring exclusively on the pinned CPU cores, preventing context switching on application worker cores.

Mastering hardware interrupt inspection and IRQ affinity tuning in Linux provides deterministic packet processing, prevents packet drop queues, and guarantees predictable performance for demanding enterprise workloads.