How to Trace TCP Socket TCP_INFO tcpi_retrans_synack Stalls in Linux Using eBPF

High-capacity ingress gateways often experience connection establishment timeouts when outbound SYN-ACK responses fail to reach incoming clients due to asymmetric routing or return-path packet loss. The Linux kernel struct tcp_info records tcpi_retrans_synack, tracking retransmitted SYN-ACK packets dispatched by the server socket. Tracing this value with eBPF gives real-time visibility into incoming handshake failure rates.

Understanding SYN-ACK Retransmission Dynamics

When a server receives a SYN packet, it enters the TCP_SYN_RECV state and dispatches a SYN-ACK. If the client fails to return the concluding ACK within the timeout interval, the kernel retransmits the SYN-ACK according to net.ipv4.tcp_synack_retries. Tracking these events isolates return-path routing issues before connections are dropped.

eBPF BCC Program for SYN-ACK Retransmission Tracing

The following BCC Python script attaches to inet_rtx_syn_ack to trace SYN-ACK drops in real time:

from bcc import BPF

bpf_source = """
#include <net/sock.h>
#include <linux/tcp.h>

struct synack_stall_event_t {
    u32 pid;
    u16 sport;
    u16 dport;
    u32 saddr;
    u32 daddr;
};

BPF_PERF_OUTPUT(synack_stall_events);

int trace_synack_stall(struct pt_regs *ctx, const struct sock *sk) {
    if (sk->sk_family != AF_INET)
        return 0;

    struct synack_stall_event_t evt = {};
    evt.pid = bpf_get_current_pid_tgid() >> 32;
    evt.sport = sk->sk_num;
    evt.dport = bpf_ntohs(sk->sk_dport);
    evt.saddr = sk->sk_rcv_saddr;
    evt.daddr = sk->sk_daddr;

    synack_stall_events.perf_submit(ctx, &evt, sizeof(evt));
    return 0;
}
"""

b = BPF(text=bpf_source)
b.attach_kprobe(event="inet_rtx_syn_ack", fn_name="trace_synack_stall")

def print_synack_stall(cpu, data, size):
    e = b["synack_stall_events"].event(data)
    print(f"SYN-ACK STALL: Server port {e.sport} -> Client {e.dport} handshake stall!")

b["synack_stall_events"].open_perf_buffer(print_synack_stall)
print("Listening for TCP SYN-ACK handshake stalls... Press Ctrl+C to terminate.")
while True:
    b.perf_buffer_poll()

Operational Diagnosis and Network Tuning

Run the script under root privileges:

sudo python3 trace_synack_stall.py

High SYN-ACK retransmissions point to asymmetric return-path firewall filtering or client-side NAT gateway drops.