How to Trace TCP Socket TCP_SYNCNT Retransmission Probing in Linux Using eBPF

When high-availability client applications establish outbound network connections across microservices or cloud APIs, connection timeouts must fail fast rather than stalling for default system thresholds (which typically retry SYN packets up to 6 times, consuming over 120 seconds). The Linux kernel provides the TCP_SYNCNT socket option via setsockopt(), allowing applications to cap the maximum number of SYN retransmission attempts before declaring connection failure (ETIMEDOUT). Tracing SYN retransmission counts and timeout aborts with eBPF gives systems architects deep visibility into dependency failures.

Mechanics of SYN Retransmission Counting

By default, the Linux kernel determines outbound SYN retransmissions using the system-wide sysctl net.ipv4.tcp_syn_retries (typically set to 6). When an application configures TCP_SYNCNT:

/* Cap SYN retransmissions to 2 attempts (failing fast in ~3s) */
int syn_retries = 2;
setsockopt(sock_fd, IPPROTO_TCP, TCP_SYNCNT, &syn_retries, sizeof(syn_retries));

The kernel overrides the global setting for that specific socket (tp->syn_retries). If remote endpoints do not answer with SYN-ACK, the kernel ceases retransmission after the specified count and returns immediate failure to the userland connect() caller.

Authoring a Custom bpftrace SYNCNT Telemetry Script

To inspect active SYN retransmission attempts and trace connections aborted due to SYN count limits, write an eBPF script using bpftrace:

#!/usr/bin/env bpftrace

#include <net/sock.h>
#include <linux/tcp.h>
#include <net/inet_connection_sock.h>

kprobe:tcp_connect
{
  $sk = (struct sock *)arg0;
  $tp = (struct tcp_sock *)arg0;

  if ($tp->syn_retries > 0) {
    @custom_syncnt_sockets[$sk] = $tp->syn_retries;
  }
}

kprobe:tcp_retransmit_timer
{
  $sk = (struct sock *)arg0;
  $icsk = (struct inet_connection_sock *)arg0;

  if ($sk->sk_state == 2) { /* TCP_SYN_SENT */
    @syn_retransmits[$sk] = $icsk->icsk_retransmits;
    time("%H:%M:%S: SYN retry %u fired on sock %p\n", 
         $icsk->icsk_retransmits, $sk);
  }
}

interval:s:10
{
  print(@custom_syncnt_sockets);
  print(@syn_retransmits);
  clear(@custom_syncnt_sockets);
  clear(@syn_retransmits);
}

Save this script as trace_syncnt.bt. It inspects outgoing connection requests and logs active SYN retries during the TCP_SYN_SENT state.

Executing and Evaluating Kernel Telemetry

Run the bpftrace monitoring script with administrative privileges:

sudo bpftrace trace_syncnt.bt

Trigger network requests to unreachable or partitioned backends. Review the telemetry metrics:

• SYN retry distribution: Ensure that @syn_retransmits does not exceed the threshold configured in @custom_syncnt_sockets.
• Fail-fast latency: Verify that connection timeouts abort within 3 to 7 seconds rather than stalling worker threads for 2 minutes.
• System-wide counters: Check nstat -z | grep -i synretrans to inspect global kernel counters.

Configuring Host Parameters for Resilient Outbound Connections

To optimize outbound connection timeouts across production services:

1. Configure TCP_SYNCNT on non-blocking HTTP clients and database pool connection sockets.
2. Inspect system defaults via sysctl net.ipv4.tcp_syn_retries.
3. Ensure firewall state tables do not drop RST responses sent upon client aborts.