Measuring initial SYN retry frequency across outbound client connections provides crucial operational insight into remote endpoint reachability and firewall drops. The Linux kernel struct tcp_info records initial handshake retransmissions under tcpi_retrans_syn. Tracing this value with eBPF gives real-time visibility into handshake retry counts before connection pool timeouts trigger application exceptions.
Understanding SYN Retry Backoff Dynamics
When a client initiates a connection via connect(), it dispatches an initial SYN packet. If no SYN-ACK arrives within 1 second, the kernel begins exponential backoff retries governed by net.ipv4.tcp_syn_retries. Tracking these retries exposes transient upstream routing failures before connection timeouts affect users.
eBPF BCC Program for SYN Retry Tracing
The following BCC Python script attaches to tcp_v4_connect and tcp_retransmit_skb to capture SYN retry counts:
from bcc import BPF
bpf_source = """
#include <net/sock.h>
#include <linux/tcp.h>
struct syn_retry_event_t {
u32 pid;
u16 sport;
u16 dport;
u32 retries;
};
BPF_PERF_OUTPUT(syn_retry_events);
int trace_retry(struct pt_regs *ctx, struct sock *sk) {
if (sk->sk_family != AF_INET)
return 0;
struct tcp_sock *tp = tcp_sk(sk);
if (sk->sk_state == TCP_SYN_SENT) {
struct syn_retry_event_t evt = {};
evt.pid = bpf_get_current_pid_tgid() >> 32;
evt.sport = sk->sk_num;
evt.dport = bpf_ntohs(sk->sk_dport);
evt.retries = tp->total_retrans;
syn_retry_events.perf_submit(ctx, &evt, sizeof(evt));
}
return 0;
}
"""
b = BPF(text=bpf_source)
b.attach_kprobe(event="tcp_retransmit_skb", fn_name="trace_retry")
def print_retry(cpu, data, size):
e = b["syn_retry_events"].event(data)
print(f"SYN RETRY: PID {e.pid:<6} | Port {e.sport} -> {e.dport} (Attempt {e.retries})")
b["syn_retry_events"].open_perf_buffer(print_retry)
print("Listening for TCP SYN retries... Press Ctrl+C to terminate.")
while True:
b.perf_buffer_poll()
Operational Diagnosis and Network Tuning
Run the script under root privileges:
sudo python3 trace_syn_retry.py
High SYN retry counts indicate that intermediate security appliances are dropping TCP handshakes or remote listening ports are firewalled.