How to Trace TCP Socket Demuxing and Ingress Lookup Latency in Linux Using eBPF

When high-speed network interface cards (NICs) process hundreds of thousands of incoming packets per second, the Linux kernel network stack must match each incoming packet to its corresponding open socket (a process termed socket demuxing). To perform this mapping, the TCP stack executes hash table lookups inside __inet_lookup_skb() querying an established socket hash table (ehash) or a listening socket hash table (lhash). In extreme connection density environments—such as API gateways or reverse proxies managing 500,000 concurrent sockets—hash bucket collisions and deep chain traversals can introduce packet processing latency. Using eBPF, systems engineers can trace socket demuxing latency and identify hash chain bottlenecks.

Understanding TCP Ingress Socket Lookup and Early Demux

The Linux kernel optimizes ingress packet routing through two complementary mechanisms:

• TCP Early Demux: Controlled via sysctl net.ipv4.tcp_early_demux. When enabled, the kernel attempts to identify the receiving socket early in the IP layer (ip_rcv_finish()), caching the socket pointer directly in the sk_buff before full protocol stack evaluation.
• Hash Table Lookup (__inet_lookup_established): Computes a four-tuple hash (source IP/port, dest IP/port) to locate the target sock structure. If the ehash table has too few buckets relative to active socket count, hash chains lengthen, consuming excessive CPU cycles inside spinlocks.

Tracing Socket Lookup Latency with bpftrace

Deploy a real-time eBPF script targeting __inet_lookup_established() to measure lookup duration and hash chain traversal depth:

# Install bpftrace on Ubuntu/Debian
sudo apt-get update && sudo apt-get install -y bpftrace linux-headers-$(uname -r)

# Trace established TCP socket lookup latency
sudo bpftrace -e '
kprobe:__inet_lookup_established {
    @start[tid] = nsecs;
}
kretprobe:__inet_lookup_established /@start[tid]/ {
    $dur_ns = nsecs - @start[tid];
    @lookup_latency_ns = hist($dur_ns);
    if ($dur_ns > 5000) { // Lookups exceeding 5 microseconds
        @slow_lookups = count();
    }
    delete(@start[tid]);
}'

In a healthy networking environment, socket lookups complete within 50 to 200 nanoseconds. Bimodal distributions with spikes exceeding 5 microseconds indicate severe ehash table chain collisions.

Inspecting System-Level Hash Bucket Allocations

Inspect system-level TCP established hash table sizing in dmesg:

# Check active ehash and lhash allocation logged during boot
dmesg | grep -E "TCP established hash table|TCP bind hash table"

# Sample output:
# TCP established hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc)
# TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)

Tuning Socket Hash Tables for High Concurrency

To eliminate socket lookup latency on high-density servers:

1. Expand ehash Buckets at Boot: Pass the thash_entries kernel boot parameter via GRUB (e.g. thash_entries=1048576) to double the available hash buckets, ensuring average hash chain depth remains close to 1.0.
2. Verify Early Demux is Active: Ensure TCP early demux is enabled via sysctl net.ipv4.tcp_early_demux=1.
3. Utilize SO_REUSEPORT with eBPF: For listening sockets, use eBPF reuseport selectors to bypass lock contention on the global listening socket table entirely.