Modern enterprise servers rely on multi-queue network interface cards (NICs) supporting Receive Side Scaling (RSS) and multi-queue transmit (TX) engines. By distributing network packets across multiple hardware queues—each serviced by a dedicated CPU core via MSI-X interrupts—Linux systems achieve 40GbE, 100GbE, and 200GbE line rates without saturating a single core. However, imperfect hashing algorithms, pinned softirq affinities, or heavy single-flow connections can cause severe queue imbalance where one hardware queue drops packets while others sit idle. The BCC eBPF tool netqtop provides real-time per-queue packet throughput diagnostics.
Understanding Multi-Queue Networking and RSS Hashing
When packets arrive at a physical network port, the NIC hardware computes a Toeplitz or symmetric hash over the packet headers (the 4-tuple: source IP, destination IP, source port, destination port). The resulting hash maps the packet to a designated RX hardware queue:
- Receive Side Scaling (RSS): Hardware distributes incoming flows evenly across available RX descriptor rings.
- Transmit Packet Steering (XPS): Kernel maps outbound socket transmissions to specific TX queues based on the sending CPU core to maximize cache locality.
- Queue Skew: If thousands of connections share identical IP/port combinations (such as behind reverse proxies, VPN concentrators, or single large database replications), hash polarization directs all traffic into a single queue, leading to ring buffer overflows and softirq stalls.
Installing the BCC Tools Suite
The netqtop utility is distributed as part of the BPF Compiler Collection (BCC):
# Ubuntu / Debian
sudo apt-get update
sudo apt-get install -y bpfcc-tools linux-headers-$(uname -r)
# RHEL 9 / Rocky Linux 9 / AlmaLinux
sudo dnf install -y bcc-tools kernel-devel-$(uname -r)
# Verify executable path
which netqtop || which /usr/share/bcc/tools/netqtop
Running netqtop to Profile Active Hardware Queues
Launch netqtop specifying the target network interface using the -n flag:
# Monitor eth0 queues with a 1-second refresh interval
sudo /usr/share/bcc/tools/netqtop -n eth0
# Monitor with custom interval and display in megabytes
sudo /usr/share/bcc/tools/netqtop -n eth0 -i 2
The tool outputs an interactive, top-like tabulated terminal dashboard:
Thu Nov 11 05:32:14 2026
QueueID BPS(RX) PPS(RX) BPS(TX) PPS(TX)
0 14.2 MB 11240 8.1 MB 6400
1 13.9 MB 10980 7.9 MB 6250
2 842.1 MB 642100 120.4 MB 92000
3 14.1 MB 11100 8.0 MB 6310
Total 884.3 MB 675420 144.4 MB 110960
Interpreting Queue Imbalances and Hotspots
The output above demonstrates an acute queue imbalance hotspot:
- Queues 0, 1, and 3 handle a modest ~14 MB/s and ~11,000 packets per second.
- Queue 2 is severely congested, handling 842 MB/s and 642,000 packets per second—accounting for over 95% of total ingress bandwidth.
- If Queue 2 exhausts its hardware descriptor ring, packets mapped to Queue 2 will be discarded, even though total system capacity is nowhere near saturation.
Resolving RSS Imbalances via ethtool Hashing Configuration
When netqtop exposes queue skew, investigate and tune NIC hashing rules:
- Inspect current RSS hashing algorithms:
sudo ethtool -k eth0 | grep rx-hashing sudo ethtool -n eth0 rx-flow-hash tcp4 - Ensure both IP addresses and TCP port numbers are factored into hash calculations (4-tuple hashing):
# Enable IP and Port hashing for IPv4 TCP sudo ethtool -N eth0 rx-flow-hash tcp4 sdfn # Enable IP and Port hashing for IPv4 UDP sudo ethtool -N eth0 rx-flow-hash udp4 sdfn - Inspect interrupt CPU core bindings:
Verify thatcat /proc/interrupts | grep -E "eth0|mlx|i40e"irqbalanceis distributing queue interrupts across distinct CPU physical cores rather than piling interrupts onto core 0.
By auditing per-queue throughput with netqtop, systems engineers can detect hash skew, tune hardware queues, and unlock the true multithreaded capacity of enterprise Linux networking hardware.