How to Trace TCP Socket TCP_INFO tcpi_retrans_synack Stalls in Linux Using eBPF

Server-side TCP handshake stalls represent one of the most frustrating performance anomalies in distributed Linux infrastructure. When a server receives an initial SYN packet, it responds with a SYN-ACK and places the half-open connection into the SYN backlog queue. If the client fails to return the completing ACK within the expected round-trip time, the kernel increments the tcpi_retrans_synack counter. By authoring targeted extended Berkeley Packet Filter (eBPF) diagnostic probes, network engineers can isolate and quantify SYN-ACK retransmission stalls in real time.

The Physiology of SYN-ACK Stalls in the Linux Kernel

Within the Linux networking subsystem, incoming connection requests enter tcp_v4_conn_request() or tcp_v6_conn_request(), creating a lightweight request_sock. The kernel then invokes inet_rtx_syn_ack() whenever the SYN-ACK retransmission timer expires. Prolonged stalls typically point to asymmetric network routing failures, intermediate stateful firewall drops, or client-side SYN-ACK packet drops caused by MTU black holes.

Constructing an eBPF Trace Program with bpftrace

The following bpftrace one-liner and script structure attaches directly to the kernel’s SYN-ACK retransmission handler, extracting the client IP, port, and retry counter:

#!/usr/bin/env bpftrace

#include <net/sock.h>
#include <net/inet_connection_sock.h>

kprobe:inet_rtx_syn_ack
{
  $sk = (struct sock *)arg0;
  $req = (struct request_sock *)arg1;
  
  $daddr = ntop($req->ir_rmt_addr);
  $dport = bswap($req->ir_rmt_port);
  $saddr = ntop($req->ir_loc_addr);
  $sport = bswap($req->ir_loc_port);
  $retries = $req->num_retrans;

  printf("[%s] SYN-ACK RTX: %s:%d -> %s:%d (Attempt: %d)
",
    strftime("%H:%M:%S", nsecs), $saddr, $sport, $daddr, $dport, $retries);

  @synack_stalls_by_client[$daddr] = count();
  @stall_distribution = hist($retries);
}

interval:s:15
{
  print(@synack_stalls_by_client);
  print(@stall_distribution);
}

Executing and Analysing Kernel Trace Streams

Save the program as synack_tracer.bt and execute it with administrative root capabilities:

sudo bpftrace synack_tracer.bt

When an upstream network segment experiences route flapping or stateful firewall discards, the tracer reveals localized clusters:

[16:42:15] SYN-ACK RTX: 10.0.4.12:443 -> 203.0.113.88:51240 (Attempt: 2)
[16:42:18] SYN-ACK RTX: 10.0.4.12:443 -> 203.0.113.88:51240 (Attempt: 3)
@synack_stalls_by_client[203.0.113.88]: 14

Operational Mitigation and Kernel Tuning

To prevent malicious SYN flood attacks from exhausting kernel resources during SYN-ACK stall conditions:

• TCP SYN Cookies: Verify that net.ipv4.tcp_syncookies = 1 is active in /etc/sysctl.conf.
• SYN-ACK Retry Limits: Tune net.ipv4.tcp_synack_retries (defaulting to 5) down to 2 or 3 in public-facing reverse proxy gateways to reclaim half-open socket memory rapidly.
• Path MTU Discovery: Ensure upstream routers do not drop ICMP Fragmentation Needed (Type 3, Code 4) packets.