How to Trace TCP Socket TCP_INFO tcpi_total_retrans Accumulation in Linux Using eBPF

Measuring lifetime packet loss accumulation across persistent long-lived database and gRPC connections requires monitoring cumulative transport counters. The Linux kernel struct tcp_info records tcpi_total_retrans, tallying the total number of retransmitted segments across the lifespan of a socket. Tracing this value with eBPF gives operations engineers granular visibility into link degradation over time.

Significance of Cumulative Retransmissions

While transient retransmission bursts indicate instantaneous congestion, persistent growth in tcpi_total_retrans signals chronic physical layer issues, severe MTU misconfigurations, or persistent bufferbloat along routing transit hops.

eBPF BCC Program for Total Retransmission Tracing

The following BCC Python script attaches to tcp_close and tcp_retransmit_skb to log cumulative retransmissions across socket lifespans:

from bcc import BPF

bpf_code = """
#include <net/sock.h>
#include <linux/tcp.h>

struct close_event_t {
    u32 pid;
    u16 sport;
    u16 dport;
    u32 total_retrans;
};

BPF_PERF_OUTPUT(close_events);

int trace_close(struct pt_regs *ctx, struct sock *sk) {
    if (sk->sk_family != AF_INET)
        return 0;

    struct tcp_sock *tp = tcp_sk(sk);
    struct close_event_t evt = {};

    evt.pid = bpf_get_current_pid_tgid() >> 32;
    evt.sport = sk->sk_num;
    evt.dport = bpf_ntohs(sk->sk_dport);
    evt.total_retrans = tp->total_retrans;

    close_events.perf_submit(ctx, &evt, sizeof(evt));
    return 0;
}
"""

b = BPF(text=bpf_code)
b.attach_kprobe(event="tcp_close", fn_name="trace_close")

def print_close(cpu, data, size):
    e = b["close_events"].event(data)
    print(f"CLOSED: PID {e.pid:<6} | Port {e.sport} -> {e.dport} | Lifetime Retrans: {e.total_retrans}")

b["close_events"].open_perf_buffer(print_close)
print("Listening for TCP connection termination metrics... Press Ctrl+C to terminate.")
while True:
    b.perf_buffer_poll()

Operational Diagnosis and Network Tuning

Run the script under root privileges:

sudo python3 trace_total_retrans.py

Sockets reporting high lifetime retransmission tallies should be correlated with specific routing paths and gateway interfaces using ip route get <dest>.