How to Trace TCP Socket TCP_INFO tcpi_sndbuf_limited in Linux Using eBPF

In high-throughput Linux network infrastructure, connections can experience transmission bottlenecks that originate not from peer flow control or network path loss, but from local kernel socket memory constraints. When an application generates data faster than the TCP stack can transmit it, or when auto-tuning fails to expand the socket write buffer (sk_sndbuf), the kernel stops accepting new writes from userspace. Applications calling write() or send() block or receive EAGAIN/EWOULDBLOCK errors, halting throughput.

The Linux kernel tracks the cumulative microseconds that a connection was stalled due to local send buffer exhaustion in struct tcp_info as tcpi_sndbuf_limited. Using extended Berkeley Packet Filter (eBPF) and bpftrace, systems engineers can trace tcpi_sndbuf_limited to diagnose socket buffer shortages in real time.

Understanding tcpi_sndbuf_limited in Kernel Space

Inside the Linux networking subsystem (defined in include/uapi/linux/tcp.h), struct tcp_info exposes send buffer stall timers:

struct tcp_info {
    /* ... state fields ... */
    __u64   tcpi_busy_time;      /* Time (us) packets were in flight */
    __u64   tcpi_rwnd_limited;   /* Time (us) stalled by peer receive window */
    __u64   tcpi_sndbuf_limited; /* Time (us) stalled by local send buffer */
    __u32   tcpi_data_segs_out;
};

When an application has data ready to send, the congestion window and peer receive window allow transmission, but the socket cannot allocate additional memory for new sk_buff structures because sk->sk_wmem_queued >= sk->sk_sndbuf, the kernel increments tp->sndbuf_limited. High tcpi_sndbuf_limited indicates that the local server’s TCP write buffer limits are too restrictive.

Authoring the bpftrace Send Buffer Stall Probe

The following bpftrace script monitors tcp_close() to audit terminated sockets, flagging flows throttled by local buffer constraints:

#!/usr/bin/env bpftrace

#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_close
{
    $sk = (struct sock *)arg0;
    $inet = (struct inet_sock *)$sk;
    $tp = (struct tcp_sock *)$sk;

    $daddr = ntop($inet->inet_daddr);
    $saddr = ntop($inet->inet_saddr);
    $dport = bswap($inet->inet_dport);
    $busy_ms = $tp->busy_time / 1000;
    $sndbuf_lim_ms = $tp->sndbuf_limited / 1000;

    if ($busy_ms > 500) {
        $stall_pct = ($sndbuf_lim_ms * 100) / $busy_ms;
        if ($stall_pct > 20) {
            time("%H:%M:%S ");
            printf("HOST BUFFER STALL | %s -> %s:%d | Busy: %d ms | SndBuf Stalled: %d ms (%d%%) | SndBuf: %d\n",
                $saddr, $daddr, $dport, $busy_ms, $sndbuf_lim_ms, $stall_pct, $sk->sk_sndbuf);
        }
    }
}

Running the Trace and Diagnosing Host Bottlenecks

Execute the bpftrace script on your Linux database server or storage worker:

sudo bpftrace trace_sndbuf_limited.bt

When socket memory is insufficient, bpftrace outputs structured telemetry:

07:15:10 HOST BUFFER STALL | 10.0.1.18 -> 198.51.100.22:443 | Busy: 6400 ms | SndBuf Stalled: 4800 ms (75%) | SndBuf: 131072
07:15:25 HOST BUFFER STALL | 10.0.1.18 -> 203.0.113.44:443 | Busy: 9200 ms | SndBuf Stalled: 6900 ms (75%) | SndBuf: 131072

In this trace, 75% of connection busy time was spent waiting for socket memory to clear, confirming that the hard limit of 128KB (131072) was artificially capping performance.

Production Tuning and Remediation

To eliminate host send buffer bottlenecks:

1. Increase default and maximum TCP write buffers in sysctl:

   sudo sysctl -w net.ipv4.tcp_wmem="4096 65536 16777216"

   Setting the maximum to 16MB allows Linux TCP auto-tuning to expand buffers to fit the bandwidth-delay product (BDP).
2. Ensure net.ipv4.tcp_moderate_rcvbuf = 1 is enabled to support dynamic auto-tuning.
3. Avoid hardcoding fixed SO_SNDBUF values in application code (such as Node.js or Java), which overrides kernel auto-tuning and freezes buffer sizes.