In high-speed networks and multi-path cloud environments, packets traversing asymmetric routing topologies can arrive at their destination out of sequence. While the Linux TCP networking stack is designed to buffer and reorder out-of-order packets inside the socket reassembly queue, severe packet reordering can deceive sender congestion control algorithms into assuming packet loss. This causes unwanted congestion window collapses, unnecessary retransmissions, and throughput throttling. The tcpcong utility traces TCP congestion events and out-of-order queue growth in real time.
How Out-of-Order Packets Trigger Congestion Control
When the Linux receiver receives a TCP segment whose sequence number is higher than the expected next sequence byte, it immediately buffers the segment in the Out-Of-Order (OFOM) queue and generates a duplicate acknowledgement (dupACK). When the sender receives duplicate ACKs, standard TCP algorithms (such as NewReno or early CUBIC implementations) assume a packet was dropped in transit, entering Fast Recovery and halving the congestion window (cwnd), even though the packet was merely delayed.
Installing Kernel Tracing Tools
Ensure that BCC tools and Linux development headers are installed on your system:
# On Debian, Ubuntu, or Pop!_OS:
sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)
# On RHEL, Rocky Linux, or Fedora:
sudo dnf install bcc-tools kernel-devel-$(uname -r)
On Debian/Ubuntu environments, execute the tool as tcpcong-bpfcc; on enterprise RHEL distributions, it is located at /usr/share/bcc/tools/tcpcong.
Running tcpcong to Monitor Congestion and Reassembly
To inspect TCP congestion events and out-of-order reassembly in real time, run tcpcong with root privileges:
sudo tcpcong-bpfcc -u
The -u flag displays socket buffer metrics in microseconds, providing fine-grained resolution on how long segments remain queued before reordering completes.
Example terminal output:
TIME(s) COMM PID LADDR:LPORT RADDR:RPORT CONG_STATE OFO_SEGS
0.000000 curl 1842 192.168.1.50:41204 104.21.5.12:443 RECOVERY 14
0.012400 mysqld 2104 192.168.1.50:3306 192.168.1.100:54120 LOSS 2
0.145000 nginx 3100 192.168.1.50:443 203.0.113.8:61200 CWR 8
Diagnosing Root Causes from OFO Segments
Evaluating the output provides actionable networking intelligence:
- High OFO_SEGS in RECOVERY: Confirms that connection throughput drops are triggered by packet reordering rather than physical link loss. When multiple segments arrive out of order, the connection flips to
RECOVERYorCWR(Congestion Window Reduced). - Multi-Queue NIC Hashing Issues: If high out-of-order arrivals occur between two dedicated servers on a local 10 GbE or 40 GbE link, the network interface card Receive Side Scaling (RSS) hashing algorithm may be distributing packets from the same TCP flow across different CPU cores.
Remediation Strategies for Packet Reordering
Based on tcpcong diagnostic data, apply appropriate kernel optimizations:
- Enable TCP SACK (Selective Acknowledgement): Ensure SACK is active, allowing receivers to acknowledge discontinuous packet ranges without triggering full window collapses:
sudo sysctl -w net.ipv4.tcp_sack=1 - Tune TCP Reordering Threshold: Adjust the baseline reordering metric to prevent premature Fast Retransmit triggers on jittery links:
sudo sysctl -w net.ipv4.tcp_reordering=6 - Switch to BBR Congestion Control: Unlike loss-based algorithms, BBR (Bottleneck Bandwidth and RTT) models link capacity rather than relying on packet loss or dupACKs, making it immune to throughput degradation from mild packet reordering.
Utilising tcpcong equips systems engineers with definitive transport-layer telemetry to diagnose reordering anomalies and optimize throughput across modern Linux networks.