How to Trace TCP Zero-Window Probe Latency and Receiver Stalls in Linux Using eBPF

When high-throughput network applications send data faster than a receiving peer can consume it from its socket buffer, the receiver advertises a receive window (win) of zero in its TCP acknowledgments. This zero-window advertisement halts all transmission from the sender. To prevent deadlocks when window update packets are lost in transit, the Linux kernel TCP stack engages the persist timer to transmit periodic 1-byte Zero-Window Probes (ZWP). In mission-critical database and distributed storage systems, receiver stalls cause severe pipeline delays. Utilising extended Berkeley Packet Filters (eBPF), Linux administrators can trace zero-window probe lifecycles and measure receiver stall duration directly in the kernel.

Understanding TCP Persist Timers and Probe Transmission

The Linux TCP state machine handles zero-window conditions via specialized kernel routines:

• tcp_probe_timer(): Invoked when the persist timer expires. Checks if the transmission window remains zero and schedules probe transmission.
• tcp_send_probe0(): Assembles and transmits a 1-byte payload or empty segment to elicit an immediate ACK containing the current receive window size from the remote receiver.
• Exponential Backoff: The persist timer starts at approximately 200ms and doubles with each probe up to TCP_RTO_MAX (typically 120 seconds). If the receiving application remains stalled, the connection throughput collapses completely.

Tracing Zero-Window Probes with bpftrace

Deploy a bpftrace one-liner to intercept kernel calls to tcp_send_probe0(), printing socket details and probe transmission timestamps:

sudo bpftrace -e '
#include <net/sock.h>
#include <net/inet_sock.h>

kprobe:tcp_send_probe0
{
    $sk = (struct sock *)arg0;
    $inet = (struct inet_sock *)$sk;
    $saddr = ntop($inet->inet_saddr);
    $daddr = ntop($inet->inet_daddr);
    $sport = bswap($inet->inet_sport);
    $dport = bswap($inet->inet_dport);

    printf("ZERO-WINDOW PROBE: %s (PID %d) | %s:%d -> %s:%d | Persist Timer Probe Fired
",
           comm, pid, $saddr, $sport, $daddr, $dport);
}'

When an outbound connection encounters a stalled receiver, bpftrace logs each probe transmission in real time:

ZERO-WINDOW PROBE: pg_dump (PID 28140) | 10.0.1.20:45210 -> 10.0.2.88:5432 | Persist Timer Probe Fired
ZERO-WINDOW PROBE: pg_dump (PID 28140) | 10.0.1.20:45210 -> 10.0.2.88:5432 | Persist Timer Probe Fired

Measuring Receiver Stall Duration with BCC

To quantify the exact duration that connections spend stalled waiting for receivers to reopen their window, author a Python BCC script:

from bcc import BPF
import time

bpf_code = """
#include <net/sock.h>
#include <net/tcp.h>

BPF_HASH(stall_start, struct sock *, u64);
BPF_HISTOGRAM(stall_duration_ms);

int kprobe__tcp_send_probe0(struct pt_regs *ctx, struct sock *sk) {
    u64 ts = bpf_ktime_get_ns();
    u64 *existing = stall_start.lookup(&sk);
    if (!existing) {
        stall_start.update(&sk, &ts);
    }
    return 0;
}

int kprobe__tcp_ack(struct pt_regs *ctx, struct sock *sk, struct sk_buff *skb, int flag) {
    struct tcp_sock *tp = tcp_sk(sk);
    // Check if receive window opened (snd_wnd > 0)
    if (tp->snd_wnd > 0) {
        u64 *start = stall_start.lookup(&sk);
        if (start) {
            u64 delta_ms = (bpf_ktime_get_ns() - *start) / 1000000;
            stall_duration_ms.increment(bpf_log2l(delta_ms));
            stall_start.delete(&sk);
        }
    }
    return 0;
}
"""

b = BPF(text=bpf_code)
print("Tracking TCP Zero-Window stall duration... Hit Ctrl+C to stop.")

try:
    while True:
        time.sleep(5)
        print("
--- Receiver Stall Duration (Log2 ms) ---")
        b["stall_duration_ms"].print_log2_hist("Duration (ms)")
except KeyboardInterrupt:
    print("
Tracing detached.")

Tuning Receiver Buffers and Kernel Memory

To eliminate zero-window stalls caused by receive buffer exhaustion on the receiving node:

1. Expand Maximum TCP Receive Buffer:

   sudo sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"

2. Verify TCP Window Scaling is Enabled:

   sudo sysctl -w net.ipv4.tcp_window_scaling=1

   Window scaling allows the receiver window to scale up to 1 gigabyte, preventing buffer bottlenecks on high-bandwidth links.
3. Application Socket Tuning: In the receiving software, ensure SO_RCVBUF is not artificially constricted, and decouple network socket reading from heavy computation using asynchronous thread worker queues.