How to Trace TCP Socket TCP_INFO tcpi_rttvar Variance in Linux Using eBPF

Diagnosing network latency jitter in microservice architectures and distributed database clusters requires monitoring not just mean round-trip time (RTT), but also RTT variance (jitter). While a steady 50ms latency across nodes is easily accommodated by TCP congestion control algorithms, high latency jitter—where packet RTTs oscillate unpredictably between 5ms and 300ms—wreaks havoc on the TCP Retransmission Timeout (RTO) estimator. When RTT variance spikes, the kernel expands its RTO timer defensively to avoid spurious retransmissions, introducing significant tail latency during packet loss recovery.

The Linux kernel tracks smoothed round-trip time and its mean deviation in struct tcp_info via tcpi_rtt and tcpi_rttvar. Using extended Berkeley Packet Filter (eBPF) and bpftrace, systems engineers can trace tcpi_rttvar directly within the kernel networking subsystem to identify jittery network paths before application timeouts occur.

Understanding RTT Variance in the Linux Kernel

The TCP protocol implements Jacobson’s algorithm for dynamic RTO estimation, defined as:

RTO = SRTT + 4 * RTTVAR

Inside the Linux kernel (specifically include/uapi/linux/tcp.h), these values are exposed through struct tcp_info in microseconds:

struct tcp_info {
    /* ... state fields ... */
    __u32   tcpi_rtt;       /* Smoothed Round Trip Time (us) */
    __u32   tcpi_rttvar;    /* RTT Variance / Mean Deviation (us) */
    __u32   tcpi_snd_ssthresh;
    __u32   tcpi_snd_cwnd;
};

When tcpi_rttvar approaches or exceeds tcpi_rtt, the network link is exhibiting severe packet queuing, cross-traffic bursts, or unstable wireless/WAN routing.

Authoring the bpftrace Jitter Detection Script

The following bpftrace script probes the tcp_rcv_established() function to sample RTT variance as incoming acknowledgements arrive:

#!/usr/bin/env bpftrace

#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_rcv_established
{
    $sk = (struct sock *)arg0;
    $inet = (struct inet_sock *)$sk;
    $tp = (struct tcp_sock *)$sk;

    $daddr = ntop($inet->inet_daddr);
    $saddr = ntop($inet->inet_saddr);
    $dport = bswap($inet->inet_dport);

    $srtt_us = $tp->srtt_us >> 3;
    $rttvar_us = $tp->mdev_us >> 2;

    // Flag sockets where jitter exceeds 15ms (15000 us)
    if ($rttvar_us > 15000) {
        time("%H:%M:%S ");
        printf("JITTER SPIKE | %s -> %s:%d | SRTT: %d us | RTTVAR: %d us | RTO: %d ms\n",
            $saddr, $daddr, $dport, $srtt_us, $rttvar_us,
            ($srtt_us + (4 * $rttvar_us)) / 1000);
    }
}

Executing the Trace and Analyzing Jitter Spikes

Run the script with administrative privileges:

sudo bpftrace trace_rttvar.bt

When routing instability or bufferbloat occurs along the path, bpftrace outputs immediate diagnostics:

19:12:04 JITTER SPIKE | 10.0.1.18 -> 198.51.100.44:443 | SRTT: 28400 us | RTTVAR: 22100 us | RTO: 116 ms
19:12:05 JITTER SPIKE | 10.0.1.18 -> 198.51.100.44:443 | SRTT: 31200 us | RTTVAR: 34500 us | RTO: 169 ms

Notice how the calculated RTO expands from 116ms to 169ms in response to the climbing RTTVAR, directly demonstrating how path jitter inflates retransmission delays.

Network Remediation and Bufferbloat Tuning

When high tcpi_rttvar is detected:

1. Inspect intermediate network switches and routers for bufferbloat; deploy active queue management algorithms such as FQ-CoDel or CAKE.
2. Transition egress TCP congestion control algorithms from loss-based CUBIC to rate-based BBR (Bottleneck Bandwidth and RTT):

   sudo sysctl -w net.ipv4.tcp_congestion_control=bbr

3. Verify that network interface cards (NICs) have Adaptive Moderation Interrupts (ethtool -C eth0 adaptive-rx on) configured correctly to minimize queue jitter.