How to Trace TCP TLP (Tail Loss Probe) and Tail Latency in Linux Using eBPF

When packet loss occurs at the end of a TCP transaction (such as the final chunk of an HTTP response or RPC payload), standard Selective Acknowledgements (SACK) and Fast Retransmit cannot trigger because no subsequent data packets exist to generate duplicate ACKs. Without mitigation, the connection stalls until the heavy Retransmission Timeout (RTO)—often lasting 200 to 1000 milliseconds—expires. The Linux kernel mitigates this via Tail Loss Probe (TLP), scheduling an early probe packet to elicit an immediate ACK. Using eBPF, systems engineers can monitor TLP activations, identify loss clusters, and eliminate p99 tail latency spikes.

Understanding Tail Loss Probe (TLP) Mechanics

When a TCP socket enters the idle phase with unacknowledged data in flight, the Linux kernel arms a Probe Timeout (PTO) timer—typically calculated as (2 imes ext{SRTT}) (smoothed round-trip time). When the PTO expires:

• If unacknowledged data remains in the send queue, the kernel retransmits the final packet as a probe.
• The probe elicits an immediate ACK or SACK from the receiver, allowing the sender to diagnose whether packet loss occurred without waiting for an expensive RTO backoff.
• While TLP significantly trims tail latency, chronic TLP triggers indicate severe upstream switch buffer drops on transaction tails.

Tracing TLP Activations with bpftrace

Deploy a real-time eBPF probe targeting the internal kernel function tcp_send_loss_probe():

# Install bpftrace on Ubuntu/Debian
sudo apt-get update && sudo apt-get install -y bpftrace linux-headers-$(uname -r)

# Trace TLP firing events and socket round-trip times
sudo bpftrace -e '
kprobe:tcp_send_loss_probe {
    $sk = (struct sock *)arg0;
    $tp = (struct tcp_sock *)$sk;
    printf("TLP probe sent: PID %d COMM %s SADDR %s DADDR %s RTT %d us
",
           pid, comm, 
           ntop($sk->__sk_common.skc_rcv_saddr),
           ntop($sk->__sk_common.skc_daddr),
           $tp->srtt_us >> 3);
    @tlp_count[comm] = count();
}'

If high-throughput reverse proxies or database connectors demonstrate hundreds of TLP triggers per minute, intermediate queues are discarding terminal bursts.

Verifying System-Wide TLP Statistics

Inspect system-level TLP counters using nstat:

# View live Tail Loss Probe counters
nstat -az | grep -E "TCPLossProbes|TCPLossProbeRecovery"

# Sample output:
# TcpExtTCPLossProbes              8412
# TcpExtTCPLossProbeRecovery      7980

A high TCPLossProbeRecovery ratio demonstrates that TLP is operating effectively, resolving loss events before standard RTO timers can expire.

Kernel Tuning for Low-Latency Workloads

To optimise TLP and tail latency in high-concurrency cloud environments:

• Verify TLP Status: Ensure TCP Early Retransmit and loss probe logic are active via sysctl net.ipv4.tcp_early_retrans=3 (enables both TLP and partial ACK fast retransmit).
• Adjust Minimum RTO: Tune the minimum retransmission timeout via route configuration: sudo ip route change default via 192.168.1.1 rto_min 50ms to prevent excessive idle wait on low-latency private networks.
• Deploy BBR Congestion Control: Transitioning from Cubic to BBR (sysctl -w net.ipv4.tcp_congestion_control=bbr) smoothens tail pacing and diminishes bufferbloat-induced tail drops.