How to Trace TCP Out-of-Order Packets and Reassembly Latency Using tcpcong in Linux

In high-speed networks and multi-path cloud environments, packets traversing asymmetric routing topologies can arrive at their destination out of sequence. While the Linux TCP networking stack is designed to buffer and reorder out-of-order packets inside the socket reassembly queue, severe packet reordering can deceive sender congestion control algorithms into assuming packet loss. This causes unwanted congestion window collapses, unnecessary retransmissions, and throughput throttling. The tcpcong utility traces TCP congestion events and out-of-order queue growth in real time.

How Out-of-Order Packets Trigger Congestion Control

When the Linux receiver receives a TCP segment whose sequence number is higher than the expected next sequence byte, it immediately buffers the segment in the Out-Of-Order (OFOM) queue and generates a duplicate acknowledgement (dupACK). When the sender receives duplicate ACKs, standard TCP algorithms (such as NewReno or early CUBIC implementations) assume a packet was dropped in transit, entering Fast Recovery and halving the congestion window (cwnd), even though the packet was merely delayed.

Installing Kernel Tracing Tools

Ensure that BCC tools and Linux development headers are installed on your system:

# On Debian, Ubuntu, or Pop!_OS:
sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

# On RHEL, Rocky Linux, or Fedora:
sudo dnf install bcc-tools kernel-devel-$(uname -r)

On Debian/Ubuntu environments, execute the tool as tcpcong-bpfcc; on enterprise RHEL distributions, it is located at /usr/share/bcc/tools/tcpcong.

Running tcpcong to Monitor Congestion and Reassembly

To inspect TCP congestion events and out-of-order reassembly in real time, run tcpcong with root privileges:

sudo tcpcong-bpfcc -u

The -u flag displays socket buffer metrics in microseconds, providing fine-grained resolution on how long segments remain queued before reordering completes.

Example terminal output:

TIME(s)     COMM           PID    LADDR:LPORT          RADDR:RPORT          CONG_STATE   OFO_SEGS
0.000000    curl           1842   192.168.1.50:41204   104.21.5.12:443      RECOVERY     14
0.012400    mysqld         2104   192.168.1.50:3306    192.168.1.100:54120  LOSS         2
0.145000    nginx          3100   192.168.1.50:443     203.0.113.8:61200    CWR          8

Diagnosing Root Causes from OFO Segments

Evaluating the output provides actionable networking intelligence:

  • High OFO_SEGS in RECOVERY: Confirms that connection throughput drops are triggered by packet reordering rather than physical link loss. When multiple segments arrive out of order, the connection flips to RECOVERY or CWR (Congestion Window Reduced).
  • Multi-Queue NIC Hashing Issues: If high out-of-order arrivals occur between two dedicated servers on a local 10 GbE or 40 GbE link, the network interface card Receive Side Scaling (RSS) hashing algorithm may be distributing packets from the same TCP flow across different CPU cores.

Remediation Strategies for Packet Reordering

Based on tcpcong diagnostic data, apply appropriate kernel optimizations:

  • Enable TCP SACK (Selective Acknowledgement): Ensure SACK is active, allowing receivers to acknowledge discontinuous packet ranges without triggering full window collapses:
    sudo sysctl -w net.ipv4.tcp_sack=1
  • Tune TCP Reordering Threshold: Adjust the baseline reordering metric to prevent premature Fast Retransmit triggers on jittery links:
    sudo sysctl -w net.ipv4.tcp_reordering=6
  • Switch to BBR Congestion Control: Unlike loss-based algorithms, BBR (Bottleneck Bandwidth and RTT) models link capacity rather than relying on packet loss or dupACKs, making it immune to throughput degradation from mild packet reordering.

Utilising tcpcong equips systems engineers with definitive transport-layer telemetry to diagnose reordering anomalies and optimize throughput across modern Linux networks.

Get the best tech tips delivered straight to your inbox.

Join thousands of readers mastering Apple, Google, Microsoft, and Linux.