How to Trace TCP Socket TCP_INFO tcpi_reordering Variance in Linux Using eBPF

In high-speed ECMP (Equal-Cost Multi-Path) datacenter networks, packets belonging to the same TCP flow can occasionally traverse different switches, arriving out of order at the destination. The Linux kernel struct tcp_info records tcpi_reordering, tracking the maximum observed packet reordering distance in segments. Tracing this value with eBPF gives network engineers direct observability into multi-path hash polarization and link jitter.

Understanding Dynamic Reordering Metric Growth

By default, Linux assumes a packet reordering threshold of 3 duplicate ACKs before triggering fast retransmit. If tcpi_reordering detects packets arriving out of order without genuine loss, the kernel dynamically increases the reordering threshold to avoid spurious retransmissions. Tracing this metric exposes whether packet loss alarms are false positives caused by routing jitter.

eBPF BCC Program for Packet Reordering Tracing

The following BCC Python script attaches to tcp_update_reordering to capture reordering events across active sockets:

from bcc import BPF

bpf_source = """
#include <net/sock.h>
#include <linux/tcp.h>

struct reorder_event_t {
    u32 pid;
    u16 sport;
    u16 dport;
    u32 reordering;
};

BPF_PERF_OUTPUT(reorder_events);

int trace_reorder(struct pt_regs *ctx, struct sock *sk) {
    if (sk->sk_family != AF_INET)
        return 0;

    struct tcp_sock *tp = tcp_sk(sk);
    struct reorder_event_t evt = {};

    evt.pid = bpf_get_current_pid_tgid() >> 32;
    evt.sport = sk->sk_num;
    evt.dport = bpf_ntohs(sk->sk_dport);
    evt.reordering = tp->reordering;

    reorder_events.perf_submit(ctx, &evt, sizeof(evt));
    return 0;
}
"""

b = BPF(text=bpf_source)
b.attach_kprobe(event="tcp_update_reordering", fn_name="trace_reorder")

def print_reorder(cpu, data, size):
    e = b["reorder_events"].event(data)
    print(f"REORDER: PID {e.pid:<6} | Port {e.sport} -> {e.dport} | Reordering Threshold: {e.reordering} segs")

b["reorder_events"].open_perf_buffer(print_reorder)
print("Listening for TCP packet reordering adaptations... Press Ctrl+C to terminate.")
while True:
    b.perf_buffer_poll()

Operational Diagnosis and Network Tuning

Execute the monitoring probe on the host machine:

sudo python3 trace_reordering.py

If reordering values escalate past 10 segments on local datacenter links, inspect ECMP hashing algorithms on top-of-rack switches to ensure consistent layer-4 flow pinning.