How to Trace TCP TIME_WAIT Socket Recycling (tcp_tw_reuse) in Linux Using eBPF

High-concurrency microservices, reverse proxies (such as Nginx and HAProxy), and API gateways open and close thousands of outbound TCP connections per second. When an endpoint performs an active close, the socket enters the TIME_WAIT state for twice the Maximum Segment Lifetime ((2 imes ext{MSL}), typically 60 seconds) to ensure delayed packets on the network do not corrupt subsequent connections. Under intense connection velocity, accumulating tens of thousands of TIME_WAIT sockets can exhaust available ephemeral ports. While the Linux kernel provides tcp_tw_reuse to safely recycle these sockets, configuration issues can still cause handshake drops. Using eBPF, systems engineers can trace socket recycling and prevent port exhaustion.

Understanding TIME_WAIT Mechanics and tcp_tw_reuse

The Linux kernel governs TIME_WAIT socket lifecycle through several sysctl directives:

• sysctl net.ipv4.tcp_tw_reuse=1: Allows the kernel to safely reuse an existing TIME_WAIT socket for a new outbound connection if the connection uses TCP timestamps (net.ipv4.tcp_timestamps=1) and the incoming timestamp is strictly newer than the last recorded packet.
• sysctl net.ipv4.tcp_max_tw_buckets: The maximum number of TIME_WAIT sockets allowed system-wide. Exceeding this threshold forces the kernel to destroy TIME_WAIT sockets immediately and log a warning in dmesg.
• Distinction: Unlike the dangerous legacy option tcp_tw_recycle (which was permanently removed in kernel 4.12 due to breaking NAT environments), tcp_tw_reuse is completely RFC-compliant and safe for outbound client sockets.

Tracing Socket Recycling with bpftrace

Deploy a real-time eBPF script targeting tcp_twsk_unique() to measure how frequently the Linux kernel reuses TIME_WAIT sockets for outbound connections:

# Install bpftrace on Ubuntu/Debian
sudo apt-get update && sudo apt-get install -y bpftrace linux-headers-$(uname -r)

# Trace TIME_WAIT socket reuse events
sudo bpftrace -e '
kprobe:tcp_twsk_unique {
    @tw_reuse_attempts[comm] = count();
}
kretprobe:tcp_twsk_unique {
    if (retval == 1) {
        @tw_reuse_success = count();
    } else {
        @tw_reuse_denied = count();
    }
}'

If @tw_reuse_denied climbs rapidly, TCP timestamps are either disabled on the local host or being stripped by intermediate firewalls, blocking socket recycling.

Monitoring Kernel TIME_WAIT Counters with ss and nstat

Inspect system-level socket states and bucket allocations:

# View current socket states across the machine
ss -s

# Check live TIME_WAIT recycling statistics
nstat -az | grep -E "TWKilled|TCPTimeWaitOverflow|TWRecycled"

# Sample output:
# TcpExtTWRecycled                 42810
# TcpExtTWKilled                   12

Tuning Ephemeral Ports and Keepalive Pools

To eliminate TIME_WAIT pressure in production environments:

1. Expand Ephemeral Port Range: Increase available outbound ports:

   sudo sysctl -w net.ipv4.ip_local_port_range="10240 65535"

2. Ensure TCP Timestamps are Active: Verify sysctl net.ipv4.tcp_timestamps=1, which is mandatory for tcp_tw_reuse to function safely.
3. Implement Upstream HTTP Keepalives: In reverse proxies like Nginx, configure upstream connection pooling (keepalive 64;) to reuse persistent connections rather than initiating fresh TCP handshakes for every request.