How to Trace TCP Retransmissions and Packet Drops in Linux Using tcpretrans

Diagnosing network packet loss and degraded throughput across distributed microservices or database clusters can be exceptionally challenging when relying solely on aggregate interface metrics. Standard tools like ip -s link or netstat -s aggregate packet drop counters across the entire host lifetime, making it difficult to correlate specific retransmission events with particular client IP addresses, processes, or timestamps. The tcpretrans utility, part of the BCC (BPF Compiler Collection) framework, utilizes eBPF probes inside the Linux kernel network stack to capture every TCP retransmission event at the exact instant it occurs.

Understanding Kernel TCP Retransmission Mechanisms

In reliable TCP networking, when a sender transmits data packets, it expects cumulative acknowledgements (ACKs) from the receiving endpoint. If an acknowledgement is not received before the Retransmission Timeout (RTO) expires, or if multiple duplicate ACKs signal packet loss (Fast Retransmit), the Linux kernel invokes tcp_retransmit_skb() to resend the lost segment.

Retransmissions introduce substantial latency spikes because the TCP congestion window (cwnd) collapses, throttling subsequent bandwidth throughput. By attaching kprobes directly to tcp_retransmit_skb(), tcpretrans identifies the exact connection tuple, protocol state, and retransmission type without running packet captures that consume heavy disk I/O.

Prerequisites and Installation

Tracing kernel socket events requires root privileges and a Linux kernel with eBPF support enabled (kernel 4.9 or higher):

Ubuntu / Debian

sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

The executable is installed as /usr/sbin/tcpretrans-bpfcc.

RHEL / Rocky Linux / AlmaLinux

sudo dnf install bcc-tools kernel-devel-$(uname -r)

The utility is located under /usr/share/bcc/tools/tcpretrans.

Tracing TCP Retransmissions in Real Time

To capture all TCP retransmissions on your Linux server, execute the utility with superuser permissions:

sudo tcpretrans-bpfcc

Whenever a TCP segment is resent, tcpretrans streams a diagnostic log row:

TIME     PID     COMM         IP LADDR:LPORT          RADDR:RPORT          STATE
05:22:10 2419    curl         4  192.168.1.50:48214   93.184.216.34:443    ESTABLISHED
05:22:11 1102    mysqld       4  10.0.0.12:3306       10.0.0.85:51420      ESTABLISHED
05:22:14 0       swapper/1    4  192.168.1.50:22      203.0.113.8:58412    ESTABLISHED

The logged columns represent:

• TIME: Timestamp of the retransmitted segment.
• PID / COMM: The process responsible for the socket, or 0 / swapper if handled within softirq kernel context.
• IP: Internet Protocol version (4 or 6).
• LADDR:LPORT: Local endpoint IP and listening or ephemeral port.
• RADDR:RPORT: Remote endpoint IP and target port.
• STATE: TCP socket state (e.g., ESTABLISHED, SYN_SENT, CLOSE_WAIT).

Advanced Filtering and TCP Flag Options

To isolate transient network issues or identify root causes across noisy environments, tcpretrans offers several specialised runtime flags:

1. Identifying SYN Packet Drops

If new outbound connections are timing out, filter to trace only initial handshake attempts by reviewing lines where STATE shows SYN_SENT. This indicates dropped SYN packets or remote firewall filtering:

sudo tcpretrans-bpfcc -c

The -c flag prints retransmission counts per connection instead of streaming continuous rows.

2. Including High-Resolution Timestamps

To correlate network anomalies with system log entries or application traces, supply the -T flag for human-readable timestamps or -t for fractional seconds since invocation:

sudo tcpretrans-bpfcc -T

3. Tracking Retransmission Reasons (Loss vs TLP)

Modern Linux kernels employ Tail Loss Probe (TLP) to detect lost tail segments early before standard RTO timers expire. The -l flag includes the specific retransmission reason:

sudo tcpretrans-bpfcc -l

Diagnostic Analysis and Troubleshooting

• Retransmissions in SYN_SENT: Indicates routing failures, remote service unavailability, or dropped SYN packets by intermediate firewalls.
• Bursts of ESTABLISHED Retransmissions: Points to localized switch buffer exhaustion, duplex mismatches, or physical link errors on the network interface. Inspect interface drops using ethtool -S <interface> | grep -E "drop|error".
• Persistent Single-Client Retransmissions: Suggests client-side packet loss (e.g., erratic Wi-Fi connectivity or cellular transitions) rather than an infrastructure outage on the server.