How to Trace TCP Tail Loss Probe (TLP) Latency in Linux Using eBPF

In short-lived, transaction-heavy RPC flows—such as HTTP microservices, DNS-over-HTTPS queries, or Redis key-value reads—packet loss occurring at the end of a data burst (“tail loss”) is devastating to request latency. Because there are no subsequent data packets to trigger Duplicate SACK acknowledgments, traditional TCP stacks stall until the heavyweight Retransmission Timeout (RTO) timer expires (typically 200ms to 1,000ms). The Linux kernel implements Tail Loss Probe (TLP, RFC 8985) to convert RTO stalls into fast retransmissions. Tracing TLP behavior with eBPF exposes tail recovery latency.

Understanding Linux Kernel Tail Loss Probe Mechanics

The Linux TCP engine schedules proactive probing packets to detect tail packet drops:

• The TLP Timer (PTO): Upon transmitting the final packet of a write buffer, the kernel calculates a Probe Timeout (PTO), typically set to 2 * RTT (or roughly 10ms to 40ms, dramatically faster than a 200ms RTO).
• Probe Dispatch: When the PTO timer expires inside tcp_send_loss_probe(), the kernel retransmits the final unacknowledged segment or transmits new data as a probe.
• Fast Recovery Conversion: The incoming ACK acknowledging the probe provides immediate feedback: if the original segment was lost, the probe’s ACK allows immediate recovery without entering high-latency Loss states.

Tracing TLP Probe Dispatches with bpftrace

Using bpftrace, capture probe transmission events directly inside the kernel loss probe routine:

sudo bpftrace -e '
#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_send_loss_probe {
    $sk = (struct sock *)arg0;
    $inet = (struct inet_sock *)arg0;
    $daddr = ntop($inet->inet_daddr);
    $dport = bswap($inet->inet_dport);

    printf("TLP Probe Fired: target=%s:%d comm=%s pid=%d\n",
           $daddr, $dport, comm, pid);
    @tlp_probes_by_peer[$daddr] = count();
}'

This script logs every socket event where Linux dispatched a Tail Loss Probe, identifying remote endpoints prone to tail burst drops.

Measuring RTO Prevention Efficiency

To inspect how many tail loss probes successfully prevent standard RTO timer timeouts:

sudo bpftrace -e '
kprobe:tcp_retransmit_timer {
    @catastrophic_rto_timeouts = count();
}

kprobe:tcp_send_loss_probe {
    @tlp_dispatches = count();
}

interval:s:10 {
    print(@tlp_probes_by_peer);
    print(@tlp_dispatches);
    print(@catastrophic_rto_timeouts);
    clear(@tlp_probes_by_peer);
}'

A high ratio of TLP dispatches compared to minimal RTO timeouts confirms that Tail Loss Probing is actively preserving sub-50ms tail latency across production API fleets.