How to Trace TCP Socket TCP_INFO tcpi_sndbuf_limited Latency in Linux Using eBPF

High-throughput data streaming services frequently suffer throughput bottlenecks when local socket send buffers fill faster than transport layers can transmit. The Linux kernel struct tcp_info records tcpi_sndbuf_limited, tallying time spent waiting for send buffer memory allocation. Tracing this value with eBPF gives systems architects instant visibility into application write stalls.

Understanding Send Buffer Pressure

When an application issues frequent write() or send() system calls on a socket with insufficient SO_SNDBUF headroom, the thread either blocks (in synchronous mode) or receives EAGAIN/EWOULDBLOCK (in asynchronous non-blocking mode). tcpi_sndbuf_limited records the cumulative microseconds a connection remained choked by local socket memory limits.

eBPF BCC Script for Buffer Starvation Tracing

The following BCC Python script attaches to tcp_check_probe_timer to detect when sockets experience send buffer exhaustion:

from bcc import BPF

bpf_source = """
#include <net/sock.h>
#include <linux/tcp.h>

struct sndbuf_event_t {
    u32 pid;
    u16 sport;
    u16 dport;
    u32 sndbuf_limited;
};

BPF_PERF_OUTPUT(sndbuf_events);

int trace_sndbuf(struct pt_regs *ctx, struct sock *sk) {
    if (sk->sk_family != AF_INET)
        return 0;

    struct tcp_sock *tp = tcp_sk(sk);
    struct sndbuf_event_t evt = {};

    evt.pid = bpf_get_current_pid_tgid() >> 32;
    evt.sport = sk->sk_num;
    evt.dport = bpf_ntohs(sk->sk_dport);
    evt.sndbuf_limited = tp->sndbuf_limited;

    sndbuf_events.perf_submit(ctx, &evt, sizeof(evt));
    return 0;
}
"""

b = BPF(text=bpf_source)
b.attach_kprobe(event="tcp_notsent_lowat_check", fn_name="trace_sndbuf")

def print_sndbuf(cpu, data, size):
    e = b["sndbuf_events"].event(data)
    print(f"PID {e.pid:<6} | Port {e.sport} -> {e.dport} | sndbuf_limited: {e.sndbuf_limited} us")

b["sndbuf_events"].open_perf_buffer(print_sndbuf)
print("Listening for tcpi_sndbuf_limited stalls... Press Ctrl+C to terminate.")
while True:
    b.perf_buffer_poll()

Operational Diagnosis and Network Tuning

Execute the monitoring probe on the server:

sudo python3 trace_sndbuf.py

If send buffer stalls occur frequently, expand maximum socket write limits via sysctl net.ipv4.tcp_wmem and configure TCP_NOTSENT_LOWAT to prevent thread thrashing.