How to Trace TCP Retransmissions and Packet Drops in Linux Using tcpretrans

When enterprise network services experience intermittent throughput degradation, connection latency spikes, or sudden socket disconnects, standard diagnostic utilities like ping or netstat often fail to pinpoint the underlying cause. While cumulative kernel SNMP counters (such as TcpRetransSegs reported by nstat) confirm that retransmissions have occurred globally, they provide no contextual telemetry indicating which process, container, IP tuple, or TCP state triggered the retry event. In production Linux environments, resolving transient network degradation requires real-time kernel tracing. Using the eBPF-based tcpretrans utility from the BPF Compiler Collection (BCC), system administrators and site reliability engineers can capture every TCP retransmission event as it happens, isolating misconfigured congestion algorithms, MTU black holes, and packet loss with negligible kernel overhead.

Understanding Kernel TCP Retransmission Mechanics

TCP is a reliable stream delivery protocol that uses sequence numbers, cumulative acknowledgements, and selective acknowledgement (SACK) blocks to ensure complete delivery across untrusted networks. Understanding why retransmissions occur helps isolate the fault domain:

• Fast Retransmit: Triggered when a sender receives duplicate ACKs (typically three consecutive duplicate ACKs) indicating that out-of-order packets arrived while an intermediate segment was lost in transit. Fast retransmit bypasses the retransmission timer.
• Retransmission Timeout (RTO): Occurs when no acknowledgements return before the dynamic RTO timer expires. RTO retransmissions signify severe path degradation, complete network partition, or severe packet drop rates.
• TLP (Tail Loss Probe): A modern Linux TCP feature that schedules an early probe segment when unacknowledged data sits at the end of a transaction, avoiding lengthy RTO waits.
• Kernel Instrumentation: Traditional packet capture with tcpdump creates high CPU and buffer overhead on multi-gigabit links. In contrast, tcpretrans attaches lightweight BPF probes directly to the kernel’s tcp_retransmit_skb() internal function, capturing metrics only when a packet drop necessitates retransmission.

Installing BCC Tools and Kernel Headers

Because tcpretrans compiles BPF bytecode against running kernel data structures or uses CO-RE (Compile Once – Run Everywhere) BTF definitions, the system requires BCC tools and matching kernel development packages.

On modern Ubuntu and Debian distributions, install the standard BCC package suite:

sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

On RHEL, Rocky Linux, or Fedora installations, enable BCC packages via dnf:

sudo dnf install bcc-tools kernel-devel-$(uname -r)

Confirm the utility is installed and verify administrative access:

which tcpretrans || which /usr/sbin/tcpretrans-bpfcc

On Debian and Ubuntu, BCC utilities append the -bpfcc suffix by default. To simplify command execution, ensure /usr/sbin is in your execution $PATH.

Real-Time TCP Retransmission Tracing

To begin tracing active retransmissions across all running processes and network interfaces, execute tcpretrans with administrative privileges:

sudo tcpretrans-bpfcc

When active retransmissions occur, the utility outputs real-time tabular telemetry:

TIME     PID    IP LADDR:LPORT          RADDR:RPORT          STATE       FLOW
04:42:10 18429  4  10.0.1.15:443    --> 198.51.100.44:52114  ESTABLISHED RTO
04:42:11 18429  4  10.0.1.15:443    --> 198.51.100.44:52114  ESTABLISHED FAST_RETRANS
04:42:15 2201   6  [2001:db8::1]:80 --> [2001:db8::2]:48902 SYN_SENT    RTO

Each field exposes vital network diagnostic data:

• TIME: Wall-clock timestamp of the kernel retransmission event.
• PID: Process identifier responsible for the network socket.
• LADDR:LPORT / RADDR:RPORT: The local and remote socket addresses and port tuples.
• STATE: The internal TCP connection state (e.g. ESTABLISHED, SYN_SENT, FIN_WAIT1). A retransmit during SYN_SENT indicates connection establishment failure or firewall drops.
• FLOW: The type of recovery trigger executed by the kernel (e.g. RTO, FAST_RETRANS, or TLP).

Advanced Filtering and TCP Metric Options

In high-throughput environments, filtering tracing output prevents terminal flooding and focuses analysis on problematic endpoints or services.

To include process command names (COMM) alongside PIDs for immediate application attribution, invoke the -c flag:

sudo tcpretrans-bpfcc -c

To display millisecond timestamps rather than formatted wall-clock strings (useful when correlating with application log entries):

sudo tcpretrans-bpfcc -t

To trace only Tail Loss Probes and Fast Retransmits while excluding conventional timeout events, or to inspect raw socket structures, leverage bpftrace one-liners when bespoke filters are required:

sudo bpftrace -e 'kprobe:tcp_retransmit_skb { printf("%s retransmitted to %s\n", comm, ntop(((struct sock *)arg0)->__sk_common.skc_daddr)); }'

Correlating TCP States with Network Bottlenecks

Interpreting the TCP state column enables administrators to rapidly triage the exact nature of network faults:

1. Retransmits in SYN_SENT: Indicates that the initial handshake packet never reached the remote server, or the server’s SYN-ACK was dropped by an upstream security group, intermediate stateful firewall, or routing loop.
2. Retransmits in ESTABLISHED with RTO: Signals severe packet drop rates on active data streams. Common culprits include bufferbloat in congested WAN switches, interface tail drops, or MTU mismatches where Path MTU Discovery (PMTUD) ICMP Fragmentation Needed packets are blocked.
3. Retransmits in FIN_WAIT_1: Demonstrates that teardown packets are failing to reach the peer, often pointing to aggressive stateful firewall timeouts that drop idle TCP sessions before clean termination occurs.

Validating Kernel Mitigations and Tuning

Once problematic TCP sessions are identified, administrators can verify interface health and evaluate TCP stack parameters to alleviate bottlenecks.

Inspect physical network interface packet drop counters using ethtool and ip:

ip -s link show dev eth0
ethtool -S eth0 | grep -E "drop|miss|error"

Verify modern congestion control and SACK settings within the Linux sysctl parameters:

sysctl net.ipv4.tcp_congestion_control
sysctl net.ipv4.tcp_sack
sysctl net.ipv4.tcp_fastopen

For high-throughput, lossy WAN environments, transitioning from legacy CUBIC to Google’s BBR (Bottleneck Bandwidth and RTT) algorithm frequently prevents aggressive throughput collapse caused by sporadic packet drops:

sudo sysctl -w net.core.default_qdisc=fq
sudo sysctl -w net.ipv4.tcp_congestion_control=bbr

Running tcpretrans before and after applying queue discipline or congestion control changes provides objective, kernel-level verification of network stability improvements.