How to Trace TCP SO_INCOMING_CPU Packet Steering Latency in Linux Using eBPF

In high-concurrency multi-threaded network architectures (such as Envoy, Nginx, or multi-queue socket workers), processing network traffic across disparate CPU cores introduces severe performance penalties. When a network interface card (NIC) handles a hardware interrupt on CPU Core 2, but the user-space worker thread reading the socket runs on CPU Core 6, network data must traverse the processor interconnect (QPI/UPI). This cross-core bounce causes L1/L2 cache invalidations, increased lock contention, and latency jitter. The Linux kernel’s SO_INCOMING_CPU socket option enables worker threads to pin socket processing directly to the core handling the network interrupt. Using eBPF, systems engineers can trace CPU steering efficiency and eliminate cache thrashing.

Understanding SO_INCOMING_CPU and RPS/RSS Steering

Modern Linux networking utilizes Receive Side Scaling (RSS) in hardware and Receive Packet Steering (RPS) in software to distribute incoming packet queues across available CPU cores:

• SO_INCOMING_CPU: When queried via getsockopt(fd, SOL_SOCKET, SO_INCOMING_CPU, &cpu, &len), the kernel returns the exact CPU core index that processed the packet’s network softirq.
• Worker Thread Routing: High-performance servers spin up dedicated worker threads pinned to specific CPU cores. By matching socket distribution to SO_INCOMING_CPU, packets are handled from hardware ring buffer to application logic entirely on a single CPU core, maximizing CPU cache locality.

Tracing Cross-Core Socket Hops with bpftrace

Deploy a real-time eBPF probe targeting tcp_v4_rcv() and tcp_recvmsg() to detect cross-core CPU migration:

# Install bpftrace on Ubuntu/Debian
sudo apt-get update && sudo apt-get install -y bpftrace linux-headers-$(uname -r)

# Trace CPU core transitions between packet arrival and user-space read
sudo bpftrace -e '
kprobe:tcp_v4_rcv {
    $skb = (struct sk_buff *)arg0;
    @packet_cpu[tid] = cpu;
}
kprobe:tcp_recvmsg {
    $curr_cpu = cpu;
    if (@packet_cpu[tid] && @packet_cpu[tid] != $curr_cpu) {
        printf("Cross-core hop detected: RX_CPU %d -> APP_CPU %d (PID %d COMM %s)
",
               @packet_cpu[tid], $curr_cpu, pid, comm);
        @cross_core_hops = count();
    }
    delete(@packet_cpu[tid]);
}'

A high @cross_core_hops tally indicates that worker processes are misaligned with hardware queue interrupt vectors.

Monitoring Interface Queue Interrupt Distributions

Inspect system-level interrupt allocations across physical CPU cores:

# View live hardware NIC interrupt distributions
watch -n 1 "cat /proc/interrupts | grep -E 'eth0|mlx5'"

# Check RPS CPU bitmasks
cat /sys/class/net/eth0/queues/rx-0/rps_cpus

Tuning Worker Thread Affinity for Maximum Locality

To eliminate cross-core latency in production workloads:

1. Pin Threads to Dedicated Cores: Bind worker threads to specific cores using pthread_setaffinity_np().
2. Align Hardware IRQs: Configure irqbalance or manually pin NIC queue interrupts to matching CPU cores using SMP affinity masks:

   echo "2" | sudo tee /proc/irq/42/smp_affinity_list

3. Enable SO_REUSEPORT with eBPF Socket Selection: Attach an eBPF SO_ATTACH_REUSEPORT_EBPF program to automatically steer inbound TCP connections to the local listener thread residing on the exact CPU core processing the initial SYN frame.