How to Trace TCP Socket TCP_INFO tcpi_retrans_synack TCP SYN Queue Overflow Drops in Linux Using eBPF

When high-density web servers experience sudden bursts of incoming connection requests, the operating system’s half-open connection table (SYN queue) can rapidly reach capacity. If the backlog overflows and SYN cookies are disabled or exhausted, the Linux kernel silently drops subsequent inbound connection segments or fails to retain half-open connection context, escalating tcpi_retrans_synack socket statistics. Extended Berkeley Packet Filters (eBPF) provide non-intrusive kernel tracing to capture queue overflow events and drop bottlenecks in real time.

Linux Kernel SYN Queue Architecture

Inside the Linux network stack, an incoming SYN packet triggers socket allocation via tcp_v4_conn_request(). The kernel evaluates whether the socket’s request queue length (reqsk_queue_len) exceeds the limit defined by listen(backlog) and the system parameter net.ipv4.tcp_max_syn_backlog.

If the queue is full and cannot generate cookies, the segment is discarded. Similarly, if the application is slow to call accept(), the listen queue overflows, causing completed three-way handshakes to be reset or dropped. These stalls force client connections into exponential retransmission backoffs. Using eBPF attached to kernel drop tracepoints gives infrastructure engineers exact per-socket overflow visibility.

Authoring the Diagnostic Probe with bpftrace

Construct a dedicated bpftrace script that intercepts TCP SYN queue drops and correlates them with SYN-ACK retransmits:

#!/usr/bin/env bpftrace
/* trace_syn_queue_drops.bt - Trace Linux TCP SYN queue overflow drops */

#include <net/sock.h>
#include <linux/tcp.h>

tracepoint:tcp:tcp_retransmit_synack
{
    $sk = (struct sock *)args->skaddr;
    $saddr = ntop(args->saddr);
    $daddr = ntop(args->daddr);
    $sport = args->sport;
    $dport = args->dport;

    printf("SYN-ACK Retransmit: %s:%d -> %s:%d (Backlog Pressure)
",
           $saddr, $sport, $daddr, $dport);
    @synack_retransmits[$saddr, $daddr] = count();
}

kprobe:tcp_v4_conn_request
{
    $sk = (struct sock *)arg0;
    $inet = (struct inet_sock *)$sk;
    $port = $sk->__sk_common.skc_num;

    @syn_requests[$port] = count();
}

kprobe:tcp_v4_syn_recv_sock
{
    $sk = (struct sock *)arg0;
    if ($sk == 0) {
        @syn_queue_drops = count();
    }
}

interval:s:5
{
    time("%H:%M:%S ");
    printf("--- TCP SYN Backlog Activity (5s Window) ---
");
    print(@syn_requests);
    print(@syn_queue_drops);
    print(@synack_retransmits);
    clear(@syn_requests);
    clear(@syn_queue_drops);
    clear(@synack_retransmits);
}

Running the Trace on Production Linux Systems

Deploy the probe with root privileges on the target server:

# Install bpftrace and kernel dependencies
sudo apt-get update && sudo apt-get install -y bpftrace linux-headers-$(uname -r)

# Execute the trace
sudo bpftrace trace_syn_queue_drops.bt

During traffic bursts, monitor @syn_queue_drops and @synack_retransmits. Spikes indicate that the listen backlog is insufficiently sized for the inbound request arrival rate.

Kernel Optimization and Parameter Tuning

To mitigate SYN queue saturation, enlarge the system listen backlog ceiling and enable SYN cookies:

# Increase core listen queue maximum
sudo sysctl -w net.core.somaxconn=65535

# Increase maximum half-open SYN backlog
sudo sysctl -w net.ipv4.tcp_max_syn_backlog=65535

# Verify SYN cookies are active
sudo sysctl -w net.ipv4.tcp_syncookies=1

# Persist configuration
cat <<EOF | sudo tee /etc/sysctl.d/99-tcp-syn-tuning.conf
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.tcp_syncookies = 1
EOF
sudo sysctl --system

Pairing kernel backlog tuning with real-time eBPF monitoring ensures reliable connection establishment during sustained traffic spikes.