How to Trace TCP Selective Acknowledgment (SACK) Holes in Linux Using eBPF

When packets are lost across congested wide area networks, traditional cumulative TCP acknowledgments force transmitters to retransmit all unacknowledged segments, introducing severe pipeline stalls. TCP Selective Acknowledgment (SACK, RFC 2018) solves this by allowing receivers to acknowledge non-contiguous blocks of successfully delivered data. The sender tracks these unreceived gaps as “SACK holes” within its internal scoreboard. However, traversing large scoreboards during packet bursts can consume excessive CPU time. Tracing SACK hole processing with eBPF isolates loss recovery bottlenecks.

Understanding Linux Kernel SACK Scoreboard Mechanics

The Linux TCP implementation maintains selective acknowledgment tracking through dynamic data structures:

• SACK Blocks: TCP option headers convey up to four distinct range blocks indicating successfully received packet sequence intervals.
• The Scoreboard: The sender kernel maintains an internal red-black tree (tp->lost_out, tp->retrans_out, tp->sacked_out) inside tcp_sacktag_write_queue() to record which segments arrived and which gaps represent lost data.
• Fast Recovery Triggers: Identifying SACK holes allows the sender to selectively retransmit only missing segments without waiting for catastrophic Retransmission Timeout (RTO) timer expiries.

Tracing SACK Scoreboard Traversal in Real Time with bpftrace

Using bpftrace, monitor SACK processing duration and track missing sequence gaps:

sudo bpftrace -e '
#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_sacktag_write_queue {
    @start[tid] = nsecs;
}

kretprobe:tcp_sacktag_write_queue /@start[tid]/ {
    $dur_us = (nsecs - @start[tid]) / 1000;
    if ($dur_us > 150) {
        printf("SACK Processing Delay: comm=%s pid=%d duration_us=%d\n", comm, pid, $dur_us);
    }
    delete(@start[tid]);
}'

This script records scoreboard tagging execution times, triggering an alert whenever kernel packet traversal consumes over 150 microseconds.

Correlating SACK Retransmissions Across Remote Endpoints

To inspect which remote network peers trigger the highest volume of selective retransmissions:

sudo bpftrace -e '
#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_mark_head_lost {
    $sk = (struct sock *)arg0;
    $inet = (struct inet_sock *)arg0;
    $daddr = ntop($inet->inet_daddr);
    $dport = bswap($inet->inet_dport);

    @sack_hole_detected[$daddr, $dport] = count();
}

interval:s:10 {
    print(@sack_hole_detected);
    clear(@sack_hole_detected);
}'

Identifying endpoints with continuous SACK hole generation isolates failing intermediate routers or asymmetrical network paths, guiding traffic rerouting across WAN links.