How to Trace TCP Socket TCP_INFO tcpi_lost in Linux Using eBPF

In high-performance Linux network diagnostics, identifying packet loss before it degrades into expensive retransmission timeouts (RTO) is essential for maintaining low tail latency. When intermediate network devices drop packets, modern TCP stacks use Selective Acknowledgements (SACK) and RACK (Recent Acknowledgment) loss detection algorithms to mark specific unacknowledged segments as lost. Inside the Linux kernel, the number of segments currently judged to be lost in transit is tracked in struct tcp_info as tcpi_lost.

Tracing tcpi_lost using extended Berkeley Packet Filter (eBPF) and bpftrace gives network engineers instantaneous visibility into packet loss events, allowing them to isolate lossy network links without running continuous packet captures.

Understanding tcpi_lost in the Linux TCP Stack

Inside the Linux networking subsystem (defined in include/uapi/linux/tcp.h), struct tcp_info exposes real-time loss metrics:

struct tcp_info {
    /* ... state fields ... */
    __u32   tcpi_unacked;       /* Unacknowledged segments in flight */
    __u32   tcpi_sacked;        /* Selectively acknowledged segments */
    __u32   tcpi_lost;          /* Segments marked as lost in flight */
    __u32   tcpi_retrans;       /* Segments currently retransmitting */
    __u32   tcpi_fackets;
};

When SACK scoreboard analysis or RACK timer expiration confirms that a packet has not reached the receiver, the kernel increments tp->lost_out (exposed as tcpi_lost) and enters the TCP Fast Recovery state (TCP_CA_Recovery). Unlike cumulative counters, tcpi_lost measures instantaneous lost flight segments.

Authoring the bpftrace Instantaneous Loss Probe

The following bpftrace script probes tcp_mark_head_lost() to capture real-time loss marking events:

#!/usr/bin/env bpftrace

#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_mark_head_lost
{
    $sk = (struct sock *)arg0;
    $inet = (struct inet_sock *)$sk;
    $tp = (struct tcp_sock *)$sk;

    $daddr = ntop($inet->inet_daddr);
    $saddr = ntop($inet->inet_saddr);
    $dport = bswap($inet->inet_dport);
    $lost = $tp->lost_out;

    time("%H:%M:%S ");
    printf("PACKET LOSS MARKED | %s -> %s:%d | Lost Segments: %d | In Flight: %d | CWND: %d\n",
        $saddr, $daddr, $dport, $lost, $tp->packets_out, $tp->snd_cwnd);
}

Running the Trace and Diagnosing Network Drops

Execute the bpftrace script with root privileges:

sudo bpftrace trace_tcp_lost.bt

During active network congestion, bpftrace outputs loss notifications in real time:

02:14:05 PACKET LOSS MARKED | 10.0.1.18 -> 198.51.100.77:443 | Lost Segments: 1 | In Flight: 32 | CWND: 32
02:14:06 PACKET LOSS MARKED | 10.0.1.18 -> 198.51.100.77:443 | Lost Segments: 3 | In Flight: 31 | CWND: 16

Notice how the congestion window is halved from 32 to 16 as multiple segments are marked lost, indicating that the kernel entered Fast Recovery cleanly without triggering a full retransmission timeout.

Production Remediation Strategies

When high tcpi_lost rates are detected:

1. Audit top-of-rack switch interfaces for CRC errors, duplex mismatches, and queue buffer discards.
2. Ensure that Explicit Congestion Notification (ECN) is enabled via sysctl net.ipv4.tcp_ecn = 1 to allow routers to signal impending queue saturation before packets are dropped.
3. Verify that Forward Error Correction (FEC) is active on 25G/100G fiber transceivers to prevent bit errors from corrupting physical Ethernet frames.