How to Trace TCP Socket TCP_INFO tcpi_retrans_synack TCP Listen Backlog Tail Drops in Linux Using eBPF

High-concurrency Linux web servers handling bursts of traffic often experience connection establishment latencies when applications fail to consume completed handshakes from the listen socket quickly enough. When the accept queue reaches its capacity limit (tail saturation), incoming final ACKs from clients are discarded by the kernel, preventing sockets from transitioning to established state and escalating tcpi_retrans_synack counters. Extended Berkeley Packet Filters (eBPF) enable non-intrusive kernel tracing to isolate listen backlog tail drops in real time.

Kernel Mechanics of Listen Backlog Tail Saturation

During the final stage of the three-way handshake, the Linux kernel processes the client ACK inside tcp_check_req() and executes tcp_v4_syn_recv_sock() to create a full socket. Inside this routine, the kernel checks whether the listen backlog queue has exceeded its maximum limit:

sk_acceptq_is_full(sk) = (sk->sk_ack_backlog > sk->sk_max_ack_backlog)

If the queue is saturated, the kernel drops the incoming ACK segment. Because the client believes the connection is established while the server discards it, the server retransmits SYN-ACK segments until timeout exhaustion, incrementing tcpi_retrans_synack in struct tcp_info. Using eBPF attached to kernel drop tracepoints gives infrastructure engineers exact per-socket overflow visibility.

Authoring the Diagnostic Probe with bpftrace

Construct a dedicated bpftrace script that intercepts TCP listen backlog drops and correlates them with SYN-ACK retransmits:

#!/usr/bin/env bpftrace
/* trace_listen_backlog_drops.bt - Trace Linux TCP listen backlog tail drops */

#include <net/sock.h>
#include <linux/tcp.h>

tracepoint:tcp:tcp_retransmit_synack
{
    $sk = (struct sock *)args->skaddr;
    $saddr = ntop(args->saddr);
    $daddr = ntop(args->daddr);
    $sport = args->sport;
    $dport = args->dport;

    printf("SYN-ACK Retransmit: %s:%d -> %s:%d (Backlog Tail Drop)
",
           $saddr, $sport, $daddr, $dport);
    @synack_retransmits[$saddr, $daddr] = count();
}

kprobe:tcp_v4_syn_recv_sock
{
    $sk = (struct sock *)arg0;
    if ($sk->sk_ack_backlog > $sk->sk_max_ack_backlog) {
        $port = $sk->__sk_common.skc_num;
        @backlog_overflow_events[$port] = count();
    }
}

interval:s:5
{
    time("%H:%M:%S ");
    printf("--- TCP Listen Backlog Activity (5s Window) ---
");
    print(@backlog_overflow_events);
    print(@synack_retransmits);
    clear(@backlog_overflow_events);
    clear(@synack_retransmits);
}

Running the Trace on Production Linux Systems

Deploy the probe with root privileges on the target server:

# Install bpftrace and kernel dependencies
sudo apt-get update && sudo apt-get install -y bpftrace linux-headers-$(uname -r)

# Execute the trace
sudo bpftrace trace_listen_backlog_drops.bt

During traffic bursts, monitor @backlog_overflow_events and @synack_retransmits. Spikes indicate that the application worker threads are processing accept calls too slowly relative to inbound arrival rates.

Kernel Optimization and Parameter Tuning

To mitigate listen backlog queue saturation, enlarge the system listen backlog ceiling and enable SYN cookies:

# Increase core listen queue maximum
sudo sysctl -w net.core.somaxconn=65535

# Increase maximum half-open SYN backlog
sudo sysctl -w net.ipv4.tcp_max_syn_backlog=65535

# Verify SYN cookies are active
sudo sysctl -w net.ipv4.tcp_syncookies=1

# Persist configuration
cat <<EOF | sudo tee /etc/sysctl.d/99-tcp-backlog-tuning.conf
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.tcp_syncookies = 1
EOF
sudo sysctl --system

Pairing kernel backlog tuning with real-time eBPF monitoring ensures reliable connection establishment during sustained traffic spikes.