How to Trace TCP Retransmissions and Segment Loss in Linux Using tcpretrans

Packet loss and network congestion are among the most elusive issues to diagnose in distributed Linux server clusters. When network packets are lost in transit, the TCP protocol automatically triggers retransmissions, leading to sudden throughput drops, latency spikes, and degraded application responsiveness. While aggregate metrics from netstat -s show cumulative retransmission counters, they cannot pinpoint when loss events happen, which remote endpoints are involved, or the socket state at the time of loss. The tcpretrans utility from the BCC toolkit traces individual TCP retransmissions in real time.

How tcpretrans Detects Kernel Retransmission Events

The tcpretrans tool attaches eBPF probes to the Linux kernel function tcp_retransmit_skb(). Whenever the TCP networking stack re-sends a packet—whether due to a Retransmission Timeout (RTO) or Fast Retransmit triggered by duplicate ACKs—tcpretrans captures the event and prints detailed socket telemetry including local and remote IP addresses, port numbers, TCP state, and the network interface involved.

Installing the BCC Tracing Tools

Ensure that BCC tools and kernel development headers are installed on your Linux distribution:

# On Debian or Ubuntu:
sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

# On RHEL, AlmaLinux, Rocky Linux, or Fedora:
sudo dnf install bcc-tools kernel-devel-$(uname -r)

On Debian/Ubuntu environments, execute the utility as tcpretrans-bpfcc; on enterprise RHEL distributions, it is located at /usr/share/bcc/tools/tcpretrans.

Executing tcpretrans to Monitor Live Retransmissions

To begin tracing TCP retransmissions system-wide in real time, run tcpretrans with root privileges:

sudo tcpretrans-bpfcc -l

The -l flag includes the TCP state (e.g., ESTABLISHED, SYN_SENT, FIN_WAIT_1) for each retransmitted packet.

Example terminal output:

TIME     PID    IP LADDR:LPORT          RADDR:RPORT          STATE       TYPE
14:32:01 0      4  192.168.1.50:443     203.0.113.10:52110   ESTABLISHED RTO
14:32:04 1842   4  192.168.1.50:54200   10.0.0.15:5432       SYN_SENT    SYN
14:32:10 0      4  192.168.1.50:443     198.51.100.4:61002   ESTABLISHED FastRetransmit

Analysing Retransmission Types and Diagnosing Root Causes

Interpreting the TYPE and STATE columns allows network engineers to isolate failure modes:

  • RTO (Retransmission Timeout): Indicates severe packet loss or a complete network blackout where no acknowledgements were returned, forcing the TCP timer to expire. Common causes include duplex mismatches, saturated router buffers, or transient routing flaps.
  • SYN Retransmissions in SYN_SENT: Sockets retransmitting during initial connection setup indicate the remote server is unreachable, blocked by a firewall, or dropping connections due to a full listen queue.
  • FastRetransmit: Occurs when the sender receives three duplicate ACKs, signaling mild out-of-order packet delivery or localized packet drops over a specific link.

Isolating Loss to Specific Interfaces or Network Hops

If retransmissions consistently correlate with a specific remote IP subnet or database cluster, check MTU sizes and Path MTU Discovery (PMTUD) issues, or investigate intermediate switches for dropped frame errors using ethtool -S <interface>.

Utilising tcpretrans eliminates guesswork from network performance tuning, allowing Linux administrators to detect packet loss instantaneously and optimize transport layer throughput.

Get the best tech tips delivered straight to your inbox.

Join thousands of readers mastering Apple, Google, Microsoft, and Linux.