How to Trace TCP Socket TCP_INFO tcpi_retransmits in Linux Using eBPF

Diagnosing transient network performance anomalies in high-throughput Linux server fleets requires granular insight into kernel socket telemetry. While coarse metrics like interface-level drops or retransmit packet rates from /proc/net/snmp confirm the presence of packet loss, they cannot isolate which specific socket connections or remote endpoints are experiencing degradation. The Linux kernel maintains comprehensive connection diagnostics inside the struct tcp_info kernel structure, accessible via the TCP_INFO socket option. Among its most critical diagnostic fields is tcpi_retransmits, which tracks the number of unacknowledged retransmission timeouts currently outstanding for an active socket.

By leveraging extended Berkeley Packet Filter (eBPF) and bpftrace, systems engineers can trace tcpi_retransmits in real time directly inside kernel space. This non-invasive observability technique provides immediate identification of degraded peer routes without incurring userspace polling overhead.

Understanding tcpi_retransmits in the Linux TCP Stack

Within the Linux networking subsystem (specifically defined in include/uapi/linux/tcp.h), the struct tcp_info structure records connection health statistics updated during packet transmission, acknowledgement, and retransmission cycles:

struct tcp_info {
    __u8    tcpi_state;
    __u8    tcpi_ca_state;
    __u8    tcpi_retransmits;
    __u8    tcpi_probes;
    __u8    tcpi_backoff;
    /* ... additional pacing and window fields ... */
    __u32   tcpi_retrans;
    __u32   tcpi_lost;
};

It is important to differentiate between tcpi_retransmits and tcpi_retrans:

• tcpi_retransmits: Represents the number of consecutive timeouts (RTO events) for the current unacknowledged head-of-line data segment. Once an acknowledgement advances the sequence space, this counter resets to zero.
• tcpi_retrans: Represents the total cumulative number of retransmissions experienced over the lifetime of the socket connection.

Spikes in tcpi_retransmits indicate acute path disruption, black-holing, or extreme peer buffer saturation where multiple successive recovery probes fail.

Tracing Retransmissions with bpftrace

Using bpftrace, you can hook into kernel functions responsible for retransmission timeouts, such as tcp_retransmit_timer() or tcp_write_timer(), and extract socket parameters directly.

#!/usr/bin/env bpftrace

#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_retransmit_timer
{
    $sk = (struct sock *)arg0;
    $inet = (struct inet_sock *)$sk;
    $tp = (struct tcp_sock *)$sk;

    $daddr = ntop($inet->inet_daddr);
    $saddr = ntop($inet->inet_saddr);
    $dport = bswap($inet->inet_dport);
    $retransmits = $tp->retransmits;

    if ($retransmits > 0) {
        time("%H:%M:%S ");
        printf("SRC: %s -> DST: %s:%d | Consecutive Retransmits: %d | SRTT: %d us\n",
            $saddr, $daddr, $dport, $retransmits, $tp->srtt_us >> 3);
    }
}

Running and Analysing the eBPF Telemetry Pipeline

Execute the bpftrace script with root privileges on your Linux diagnostic node:

sudo bpftrace retransmits_probe.bt

When transient network congestion occurs, bpftrace outputs structured telemetry identifying failing connections instantly:

14:32:01 SRC: 192.168.10.45 -> DST: 10.0.4.12:443 | Consecutive Retransmits: 1 | SRTT: 24150 us
14:32:03 SRC: 192.168.10.45 -> DST: 10.0.4.12:443 | Consecutive Retransmits: 2 | SRTT: 48300 us
14:32:07 SRC: 192.168.10.45 -> DST: 10.0.4.12:443 | Consecutive Retransmits: 3 | SRTT: 96600 us

Notice the doubling of the smoothed round-trip time (SRTT) backoff calculation between successive retransmissions. This pattern reveals exponential backoff triggered by intermediate packet black-holing.

Production Hardening and Tuning

When running eBPF tracepoints in high-throughput production environments, apply filter predicates on destination ports or internal CIDR blocks to minimise kernel overhead. In addition, monitor /proc/sys/net/ipv4/tcp_retries2 to align your alerting thresholds with the operating system’s maximum socket timeout lifecycle.