How to Trace TCP Zero-Window Stalls and Receive Buffer Starvation in Linux Using eBPF

In high-throughput TCP streaming and distributed file system workloads, sender throughput can collapse abruptly even when physical network bandwidth remains abundant. This breakdown frequently occurs when the receiving host’s application layer fails to drain incoming socket buffers quickly enough. When the receive buffer (sk_rcvbuf) fills, the Linux TCP stack advertises a receive window of zero (win 0), forcing the sender to halt transmission and initiate zero-window persist probes. Using eBPF, systems engineers can trace zero-window events, identify consumer thread stalls, and tune kernel memory allocations.

Understanding TCP Zero-Window Mechanics and Window Probes

When an application’s event loop stalls due to lock contention or disk I/O bottlenecks, incoming packets remain buffered inside the kernel socket receive queue:

• When available buffer space drops below the path MSS, the kernel transmits an ACK packet with Window: 0.
• The sending host enters the TCP persist state, pausing data transfer and dispatching single-byte "zero-window probes" at exponential backoff intervals to check if buffer space has freed up.
• If the receiving application remains unresponsive, persistent zero-window conditions can lock transmission queues for hundreds of milliseconds.

Tracing Zero-Window Advertisements with bpftrace

Deploy a real-time eBPF script targeting tcp_select_window() to detect whenever the Linux kernel advertises an empty window to remote peers:

# Install bpftrace on Ubuntu/Debian
sudo apt-get update && sudo apt-get install -y bpftrace linux-headers-$(uname -r)

# Trace zero-window generation events
sudo bpftrace -e '
kretprobe:tcp_select_window {
    $win = retval;
    if ($win == 0) {
        $sk = (struct sock *)arg0;
        printf("Zero-window advertised: PID %d COMM %s SADDR %s DADDR %s RCVBUF %d
",
               pid, comm,
               ntop($sk->__sk_common.skc_rcv_saddr),
               ntop($sk->__sk_common.skc_daddr),
               $sk->sk_rcvbuf);
        @zero_win_count[comm] = count();
    }
}'

This script pinpoints the exact processes (such as ingress proxies or local log shippers) whose processing lag is choking remote transmission pipelines.

Monitoring Kernel Zero-Window Counters with nstat

Inspect system-level TCP zero-window counters to evaluate whether the local host is receiving or issuing window stalls:

# Check live TCP zero-window and probe statistics
nstat -az | grep -E "TCPZeroWindow|TCPWinProbe"

# Sample output:
# TcpExtTCPZeroWindowDrop          128
# TcpExtTCPWinProbe                4520

Tuning Socket Receive Buffers and Application Drains

To eliminate receive buffer starvation in production environments:

1. Adjust Global Receive Buffer Limits: Scale TCP receive memory boundaries in /etc/sysctl.conf:

   net.ipv4.tcp_rmem = 4096 87380 16777216
   net.core.rmem_max = 16777216

2. Enable TCP Window Scaling: Ensure RFC 1323 window scaling remains enabled via sysctl net.ipv4.tcp_window_scaling=1.
3. Audit Application Read Loops: Ensure worker threads are not blocking on synchronous database writes or blocking disk operations within the primary socket polling loop.