How to Trace TCP Zero-Window Probes and Stalls in Linux Using eBPF

Network performance degradation across enterprise Linux systems is frequently misdiagnosed as packet loss or bandwidth congestion when the actual bottleneck lies within application receive buffers. When a receiving daemon (such as an ingestion pipeline, database replica, or file server) cannot process incoming socket buffers quickly enough, its TCP receive window shrinks to zero. In response, the kernel issues a TCP Zero-Window advertisement, forcing the sender into persist mode and transmitting periodic Zero-Window Probes. Tracing these stalls using eBPF allows systems engineers to pinpoint buffer exhaustion and slow consumers in real time.

Understanding TCP Window Exhaustion and Zero-Window Probes

TCP incorporates sliding window flow control to prevent senders from overwhelming receivers. Every TCP header includes a 16-bit Window Size field multiplied by a negotiated window scale factor. This indicates how many bytes of unacknowledged data the receiving host is currently willing to accept in its socket buffer (sk_rcvbuf).

If the user-space process stalls or experiences CPU starvation, incoming packets accumulate in the kernel receive queue. Once the allocated memory limit is reached, the kernel advertises a window size of 0. The sender halts transmission and initializes a persist timer, sending single-byte Zero-Window Probes (ZWP) to poll whether buffer space has reopened. If probes continue unanswered, transfers stall completely.

Prerequisites and Tooling

Tracing kernel socket flags and probe sequences requires root privileges and a kernel supporting BPF Type Format (Linux 5.4 or higher recommended):

# Ubuntu / Debian
sudo apt update && sudo apt install bpftrace bpfcc-tools linux-headers-$(uname -r)

# RHEL / Rocky Linux
sudo dnf install bpftrace bpftrace-tools kernel-devel-$(uname -r)

Tracing Zero-Window Advertisements in Real Time

To detect when a local server sends a Zero-Window advertisement to an external peer, attach a bpftrace probe to tcp_select_window() or inspect outgoing TCP flags:

sudo bpftrace -e '
#include <net/sock.h>
#include <linux/tcp.h>

kretprobe:tcp_select_window {
    $win = (uint16)retval;
    if ($win == 0) {
        $sk = (struct sock *)arg0;
        printf("ZERO-WINDOW ADVERTISED: %s | Port: %d | Comm: %s | PID: %d
",
            time("%H:%M:%S"), $sk->__sk_common.skc_num, comm, pid);
    }
}'

When triggered, this script outputs the exact time, local port, process name, and process ID responsible for the zero-window state, immediately identifying which local service has stopped reading from its socket.

Detecting Inbound Zero-Window Probes on the Sender Side

When troubleshooting a Linux host acting as a data sender (e.g., an ETL pipeline or proxy), you can trace when the sender enters the TCP persist timer state to probe a remote zero-window peer:

sudo bpftrace -e '
#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_send_probe0 {
    $sk = (struct sock *)arg0;
    $inet = (struct inet_sock *)arg0;
    $daddr = ntop($inet->sk.__sk_common.skc_daddr);
    $dport = bswap($inet->sk.__sk_common.skc_dport);
    printf("%s [PROBE0] Sender stalled waiting for receiver %s:%d | Comm: %s
",
        time("%H:%M:%S"), $daddr, $dport, comm);
}'

If this probe triggers repeatedly, the remote receiver specified in the log has exhausted its buffer capacity and is throttling your transmission.

Diagnostic Analysis and Troubleshooting Steps

• Distinguish Network vs Application Latency: If packet captures show zero-window packets accompanied by high TCP round-trip times, the delay is caused by application processing queues rather than physical network congestion.
• Check Socket Buffer Allocations: Inspect the active socket receive buffer memory using ss -tmpi. Review the rcv_space and rcv_ssthresh metrics for the affected connection.
• Audit CPU Contention: Correlate zero-window spikes with high process CPU or garbage collection cycles using pidstat -u -p <PID> 1.

Remediation and Kernel Tuning

• Expand System Auto-Tuning Limits: Ensure Linux TCP auto-tuning is enabled and allows sufficient maximum receive memory:

  # Check max receive memory parameters (min, default, max)
  sysctl net.ipv4.tcp_rmem
  
  # Recommend increasing maximum buffer for high-bandwidth interfaces
  sudo sysctl -w net.ipv4.tcp_rmem="4096 131072 16777216"

• Application Worker Concurrency: If the application is single-threaded, implement asynchronous event loops or expand worker thread pools so consumer threads drain socket buffers concurrently.