Packet loss and network congestion are among the most elusive issues to diagnose in distributed Linux server clusters. When network packets are lost in transit, the TCP protocol automatically triggers retransmissions, leading to sudden throughput drops, latency spikes, and degraded application responsiveness. While aggregate metrics from netstat -s show cumulative retransmission counters, they cannot pinpoint when loss events happen, which remote endpoints are involved, or the socket state at the time of loss. The tcpretrans utility from the BCC toolkit traces individual TCP retransmissions in real time.
How tcpretrans Detects Kernel Retransmission Events
The tcpretrans tool attaches eBPF probes to the Linux kernel function tcp_retransmit_skb(). Whenever the TCP networking stack re-sends a packet—whether due to a Retransmission Timeout (RTO) or Fast Retransmit triggered by duplicate ACKs—tcpretrans captures the event and prints detailed socket telemetry including local and remote IP addresses, port numbers, TCP state, and the network interface involved.
Installing the BCC Tracing Tools
Ensure that BCC tools and kernel development headers are installed on your Linux distribution:
# On Debian or Ubuntu:
sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)
# On RHEL, AlmaLinux, Rocky Linux, or Fedora:
sudo dnf install bcc-tools kernel-devel-$(uname -r)
On Debian/Ubuntu environments, execute the utility as tcpretrans-bpfcc; on enterprise RHEL distributions, it is located at /usr/share/bcc/tools/tcpretrans.
Executing tcpretrans to Monitor Live Retransmissions
To begin tracing TCP retransmissions system-wide in real time, run tcpretrans with root privileges:
sudo tcpretrans-bpfcc -l
The -l flag includes the TCP state (e.g., ESTABLISHED, SYN_SENT, FIN_WAIT_1) for each retransmitted packet.
Example terminal output:
TIME PID IP LADDR:LPORT RADDR:RPORT STATE TYPE
14:32:01 0 4 192.168.1.50:443 203.0.113.10:52110 ESTABLISHED RTO
14:32:04 1842 4 192.168.1.50:54200 10.0.0.15:5432 SYN_SENT SYN
14:32:10 0 4 192.168.1.50:443 198.51.100.4:61002 ESTABLISHED FastRetransmit
Analysing Retransmission Types and Diagnosing Root Causes
Interpreting the TYPE and STATE columns allows network engineers to isolate failure modes:
- RTO (Retransmission Timeout): Indicates severe packet loss or a complete network blackout where no acknowledgements were returned, forcing the TCP timer to expire. Common causes include duplex mismatches, saturated router buffers, or transient routing flaps.
- SYN Retransmissions in SYN_SENT: Sockets retransmitting during initial connection setup indicate the remote server is unreachable, blocked by a firewall, or dropping connections due to a full listen queue.
- FastRetransmit: Occurs when the sender receives three duplicate ACKs, signaling mild out-of-order packet delivery or localized packet drops over a specific link.
Isolating Loss to Specific Interfaces or Network Hops
If retransmissions consistently correlate with a specific remote IP subnet or database cluster, check MTU sizes and Path MTU Discovery (PMTUD) issues, or investigate intermediate switches for dropped frame errors using ethtool -S <interface>.
Utilising tcpretrans eliminates guesswork from network performance tuning, allowing Linux administrators to detect packet loss instantaneously and optimize transport layer throughput.