How to Trace TCP Socket TCP_INFO tcpi_retrans_synack TCP SYN Flood Cookie Depletion in Linux Using eBPF

Under heavy SYN flood attacks or severe network congestion, Linux servers rely on SYN cookies (RFC 4987) to maintain service availability when the connection backlog queue fills up. However, when the inbound connection rate overwhelms the cookie generation pipeline or when clients fail to complete handshakes, SYN-ACK retransmission timers expire, resulting in elevated tcpi_retrans_synack counters and dropped handshakes. Extended Berkeley Packet Filters (eBPF) provide non-intrusive kernel observability to diagnose SYN cookie depletion and handshake stalls in real time.

Linux Kernel SYN Cookie Architecture

When the listen backlog for a socket fills up and net.ipv4.tcp_syncookies = 1 is active, the Linux kernel invokes cookie_v4_init_sequence() to compute a cryptographic hash containing connection state inside the initial sequence number of the SYN-ACK segment. If client ACKs arrive with valid cookie hashes, connections transition directly into ESTABLISHED state without consuming SYN table entries.

However, when flood volumes saturate CPU cryptographic hashing or when legitimate client ACKs are lost during return transit, the listening socket cannot recover the connection state, triggering repeated SYN-ACK retransmissions tracked in struct tcp_info. Tracing these events at the kernel function boundary reveals whether connection drops stem from queue exhaustion, hashing bottlenecks, or network packet loss.

Developing the bpftrace Monitoring Script

Author a targeted bpftrace probe to intercept SYN cookie generation, validation failures, and SYN-ACK retransmits:

#!/usr/bin/env bpftrace
/* trace_syn_cookie_depletion.bt - Trace Linux SYN cookie generation and drops */

#include <net/sock.h>
#include <linux/tcp.h>

kprobe:cookie_v4_init_sequence
{
    @syn_cookies_generated = count();
}

kprobe:cookie_v4_check
{
    @syn_cookies_checked = count();
}

tracepoint:tcp:tcp_retransmit_synack
{
    $sk = (struct sock *)args->skaddr;
    $saddr = ntop(args->saddr);
    $daddr = ntop(args->daddr);
    $sport = args->sport;
    $dport = args->dport;

    printf("SYN-ACK Retransmit during flood: %s:%d -> %s:%d
",
           $saddr, $sport, $daddr, $dport);
    @synack_flood_retrans[$saddr, $daddr] = count();
}

interval:s:5
{
    time("%H:%M:%S ");
    printf("--- SYN Cookie Generation vs Validation (5s) ---
");
    print(@syn_cookies_generated);
    print(@syn_cookies_checked);
    print(@synack_flood_retrans);
    clear(@syn_cookies_generated);
    clear(@syn_cookies_checked);
    clear(@synack_flood_retrans);
}

Running the Trace on Production Linux Systems

Deploy the probe with root privileges on the target server:

# Install bpftrace and kernel headers
sudo apt-get update && sudo apt-get install -y bpftrace linux-headers-$(uname -r)

# Execute the trace
sudo bpftrace trace_syn_cookie_depletion.bt

During traffic bursts, monitor the ratio of generated cookies to validated cookies. A wide divergence accompanied by high synack_flood_retrans indicates that client handshakes are failing to complete, pointing to upstream return-path filtering or client ACK drops.

Kernel Optimization and Parameter Tuning

To reinforce the kernel against backlog depletion during connection spikes:

# Increase system listen backlog maximum
sudo sysctl -w net.core.somaxconn=65535

# Enlarge TCP SYN backlog queue
sudo sysctl -w net.ipv4.tcp_max_syn_backlog=65535

# Persist settings in sysctl.d
cat <<EOF | sudo tee /etc/sysctl.d/99-syn-backlog.conf
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.tcp_syncookies = 1
EOF
sudo sysctl --system

Fine-tuning these parameters alongside continuous eBPF monitoring ensures reliable connection establishment even during sustained traffic spikes.