How to Trace TCP Socket TCP_INFO tcpi_rwnd_limited in Linux Using eBPF

In high-throughput server environments, application developers frequently blame network congestion, packet loss, or server CPU bottlenecks when client download speeds plummet. However, one of the most common causes of stalled throughput is client-side receiver exhaustion: the client operating system or application cannot process inbound data as fast as the server transmits it. When the client’s socket receive buffer fills up, its advertised receive window drops, forcing the server kernel to freeze packet transmission.

The Linux kernel tracks the cumulative microseconds that a TCP connection was completely stalled due to peer receive window limits in struct tcp_info as tcpi_rwnd_limited. Tracing tcpi_rwnd_limited using extended Berkeley Packet Filter (eBPF) and bpftrace gives network engineers definitive proof of client-side backpressure in real time.

Understanding tcpi_rwnd_limited in Kernel Space

Inside the Linux networking subsystem (defined in include/uapi/linux/tcp.h), struct tcp_info exposes transport stall timers:

struct tcp_info {
    /* ... state fields ... */
    __u64   tcpi_busy_time;      /* Time (us) packets were in flight */
    __u64   tcpi_rwnd_limited;   /* Time (us) stalled by peer receive window */
    __u64   tcpi_sndbuf_limited; /* Time (us) stalled by local send buffer */
    __u32   tcpi_snd_wnd;        /* Current advertised window (bytes) */
};

Whenever a socket has application data ready to send, but the sender’s congestion window is larger than the peer’s advertised receive window (snd_cwnd > snd_wnd) and snd_wnd is fully saturated, the kernel increments tp->rwnd_limited. If tcpi_rwnd_limited constitutes a large fraction of tcpi_busy_time, the network and server are healthy—the bottleneck is entirely on the client side.

Authoring the bpftrace Receiver Window Stall Probe

The following bpftrace script monitors tcp_close() to audit connections upon termination, identifying flows severely throttled by client backpressure:

#!/usr/bin/env bpftrace

#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_close
{
    $sk = (struct sock *)arg0;
    $inet = (struct inet_sock *)$sk;
    $tp = (struct tcp_sock *)$sk;

    $daddr = ntop($inet->inet_daddr);
    $saddr = ntop($inet->inet_saddr);
    $dport = bswap($inet->inet_dport);
    $busy_ms = $tp->busy_time / 1000;
    $rwnd_lim_ms = $tp->rwnd_limited / 1000;

    if ($busy_ms > 500) {
        $stall_pct = ($rwnd_lim_ms * 100) / $busy_ms;
        if ($stall_pct > 25) {
            time("%H:%M:%S ");
            printf("CLIENT BACKPRESSURE | %s -> %s:%d | Busy: %d ms | Rwnd Stalled: %d ms (%d%%)\n",
                $saddr, $daddr, $dport, $busy_ms, $rwnd_lim_ms, $stall_pct);
        }
    }
}

Running the Trace and Diagnosing Client Bottlenecks

Execute the bpftrace script on your Linux streaming server or API gateway:

sudo bpftrace trace_rwnd_limited.bt

When clients fail to drain receive queues, bpftrace outputs immediate diagnostics:

06:22:10 CLIENT BACKPRESSURE | 10.0.1.14 -> 198.51.100.55:443 | Busy: 8200 ms | Rwnd Stalled: 6400 ms (78%)
06:22:25 CLIENT BACKPRESSURE | 10.0.1.14 -> 203.0.113.89:443 | Busy: 15400 ms | Rwnd Stalled: 12100 ms (78%)

This telemetric proof demonstrates that over 78% of active connection time was spent idle because the remote client’s receive window was completely saturated.

Production Applications and Remediation

Tracking tcpi_rwnd_limited via eBPF helps systems teams:

1. Defend server infrastructure SLAs by demonstrating that throughput throttling originates on client endpoints.
2. Identify clients with misconfigured TCP window scaling (RFC 7323) or artificially capped socket receive buffers.
3. Tune server-side backpressure pipelines to defer database queries until client windows reopen.