How to Trace TCP Socket TCP_INFO tcpi_total_retrans in Linux Using eBPF

Tracking historical network degradation across long-lived TCP connections—such as database connection pools, gRPC streams, and real-time data replication feeds—demands visibility into cumulative socket retransmission counters. While ephemeral metrics like tcpi_retransmits monitor immediate timeout probes, they reset to zero as soon as a single acknowledgement is received. Consequently, an intermittent connection that experiences frequent, isolated packet drops over many hours may show zero active retransmits during point-in-time checks. To audit lifetime transport health, the Linux kernel provides the tcpi_total_retrans counter inside struct tcp_info, recording the absolute total number of retransmitted segments across the connection’s entire lifecycle.

Using extended Berkeley Packet Filter (eBPF) tools, systems engineers can trace tcpi_total_retrans non-invasively at the kernel level, capturing cumulative drop ratios and diagnosing silent connection degradation across production environments.

Understanding tcpi_total_retrans in Kernel Space

In modern Linux kernels (version 4.10 and newer), struct tcp_info includes dedicated 32-bit cumulative telemetry fields:

struct tcp_info {
    /* ... state and timing fields ... */
    __u32   tcpi_segs_out;       /* Total segments sent */
    __u32   tcpi_segs_in;        /* Total segments received */
    __u32   tcpi_retrans;        /* Cumulative retransmits */
    __u32   tcpi_total_retrans;  /* Total retransmitted segments */
};

By comparing tcpi_total_retrans against tcpi_segs_out, network reliability engineers can calculate the exact cumulative Packet Error Rate (PER) for any individual socket:

Packet Error Rate = (tcpi_total_retrans / tcpi_segs_out) * 100%

If this ratio exceeds 1.5% on internal low-latency infrastructure, it indicates physical link defects, misconfigured MTU jumbo frames, or buffer overrun on top-of-rack switches.

Authoring the bpftrace Socket Audit Script

The following bpftrace script probes the tcp_close() kernel function to summarise connection transport metrics as sockets terminate, pinpointing degraded flows:

#!/usr/bin/env bpftrace

#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_close
{
    $sk = (struct sock *)arg0;
    $inet = (struct inet_sock *)$sk;
    $tp = (struct tcp_sock *)$sk;

    $daddr = ntop($inet->inet_daddr);
    $saddr = ntop($inet->inet_saddr);
    $dport = bswap($inet->inet_dport);
    $total_retrans = $tp->total_retrans;
    $segs_out = $tp->segs_out;

    if ($total_retrans > 5) {
        time("%H:%M:%S ");
        printf("CLOSED %s -> %s:%d | Segs: %d | Total Retrans: %d | Drop Rate: %d%%\n",
            $saddr, $daddr, $dport, $segs_out, $total_retrans,
            $segs_out > 0 ? ($total_retrans * 100) / $segs_out : 0);
    }
}

Running the Trace and Analysing Socket Lifecycles

Execute the bpftrace script on your Linux database server or API gateway:

sudo bpftrace socket_retrans_audit.bt

The output streams connection lifecycle summaries in real time as sockets close:

16:14:02 CLOSED 10.0.1.15 -> 10.0.8.22:5432 | Segs: 182400 | Total Retrans: 12 | Drop Rate: 0%
16:15:45 CLOSED 10.0.1.15 -> 172.16.4.99:9092 | Segs: 45020 | Total Retrans: 2340 | Drop Rate: 5%

The second log entry highlights an immediate anomaly: a Kafka connection to 172.16.4.99:9092 suffered a 5% packet drop rate over its lifecycle, signalling network path degradation on that subnet.

Production Considerations and Best Practices

In high-throughput environments handling tens of thousands of connections per second, tracing tcp_close via kprobe incurs virtually no overhead because the hook fires only upon connection teardown. To monitor long-lived sockets before termination, pair this script with an interval timer hook (e.g., interval:s:60) to sample active connections periodically from tcp_hashinfo.