How to Trace TCP Socket TCP_INFO tcpi_retrans Slowdowns in Linux Using eBPF

Sporadic network retransmissions can severely degrade distributed microservice RPC latencies. The Linux kernel struct tcp_info records tcpi_retrans, representing the total number of unacknowledged packets currently undergoing retransmission. Tracing this value with eBPF exposes transient transport loss episodes in real time without capturing bulky packet dumps.

Understanding Active Retransmissions

While cumulative retransmission counters track lifetime connection errors, tcpi_retrans measures the immediate number of unacknowledged retransmitted segments outstanding in the pipe. Spikes in this metric highlight immediate queue drops in switch fabrics or localized WiFi interference.

eBPF BCC Program for Retransmission Monitoring

The following BCC Python script attaches to tcp_retransmit_skb to capture active retransmissions alongside local socket descriptors:

from bcc import BPF

bpf_source = """
#include <net/sock.h>
#include <linux/tcp.h>

struct retrans_event_t {
    u32 pid;
    u16 sport;
    u16 dport;
    u32 retrans;
};

BPF_PERF_OUTPUT(retrans_events);

int trace_retransmit(struct pt_regs *ctx, struct sock *sk) {
    if (sk->sk_family != AF_INET)
        return 0;

    struct tcp_sock *tp = tcp_sk(sk);
    struct retrans_event_t evt = {};

    evt.pid = bpf_get_current_pid_tgid() >> 32;
    evt.sport = sk->sk_num;
    evt.dport = bpf_ntohs(sk->sk_dport);
    evt.retrans = tp->retrans_out;

    retrans_events.perf_submit(ctx, &evt, sizeof(evt));
    return 0;
}
"""

b = BPF(text=bpf_source)
b.attach_kprobe(event="tcp_retransmit_skb", fn_name="trace_retransmit")

def print_retrans(cpu, data, size):
    e = b["retrans_events"].event(data)
    print(f"RETRANS: PID {e.pid:<6} | Port {e.sport} -> {e.dport} | Retrans Out: {e.retrans} packets")

b["retrans_events"].open_perf_buffer(print_retrans)
print("Listening for TCP retransmissions... Press Ctrl+C to terminate.")
while True:
    b.perf_buffer_poll()

Operational Diagnosis and Network Tuning

Execute the monitoring probe on the production host:

sudo python3 trace_retrans.py

Correlate retransmission spikes with switch port queue drops (via ethtool -S <interface>) to isolate whether packet losses stem from shallow hardware packet buffers or misconfigured ECN thresholds.