How to Trace TCP Socket TCP_INFO tcpi_notsent_lowat in Linux Using eBPF

In high-throughput HTTP/2, HTTP/3, and real-time streaming servers, application-level bufferbloat inside kernel socket write queues can severely degrade responsiveness. When an application writes large data blocks (such as a 10MB file download) into a TCP socket, the Linux kernel buffers the entire payload in the socket send buffer (sk_sndbuf). Consequently, if higher-priority packets (such as interactive HTTP/2 control frames or video audio frames) need to be transmitted, they are blocked behind megabytes of previously queued unsent data, introducing hundreds of milliseconds of head-of-line delay.

To eliminate this latency, modern Linux kernels provide the TCP_NOTSENT_LOWAT socket option, which prevents the kernel from notifying the application that the socket is writable until unsent queued data drops below a strictly configured low-water mark. Tracing tcpi_notsent_lowat telemetry using extended Berkeley Packet Filter (eBPF) allows systems engineers to audit queue bounds and eliminate server-side bufferbloat.

Understanding TCP_NOTSENT_LOWAT in Kernel Space

Inside the Linux networking subsystem (defined in include/uapi/linux/tcp.h), struct tcp_info exposes unsent buffer telemetry:

struct tcp_info {
    /* ... state fields ... */
    __u32   tcpi_notsent_bytes;  /* Bytes currently queued in send buffer not yet sent */
    __u32   tcpi_min_rtt;
    __u32   tcpi_data_segs_in;
    __u32   tcpi_data_segs_out;
};

When an application sets setsockopt(fd, IPPROTO_TCP, TCP_NOTSENT_LOWAT, &limit, sizeof(limit)) (e.g. 16KB), the kernel’s tcp_stream_memory_free() function reports that the socket is writable only when tp->write_seq - tp->snd_nxt < limit. This bounds unsent queue delays to negligible levels.

Authoring the bpftrace Unsent Buffer Probe

The following bpftrace script monitors tcp_write_xmit() to detect instances where unsent buffered bytes exceed 32KB on active sockets:

#!/usr/bin/env bpftrace

#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_write_xmit
{
    $sk = (struct sock *)arg0;
    $inet = (struct inet_sock *)$sk;
    $tp = (struct tcp_sock *)$sk;

    $daddr = ntop($inet->inet_daddr);
    $saddr = ntop($inet->inet_saddr);
    $dport = bswap($inet->inet_dport);
    $notsent = $tp->write_seq - $tp->snd_nxt;

    // Trigger alert if unsent queued data exceeds 32KB (bufferbloat condition)
    if ($notsent > 32768) {
        time("%H:%M:%S ");
        printf("BUFFERBLOAT ALERT | %s -> %s:%d | Unsent Bytes: %d | CWND: %d\n",
            $saddr, $daddr, $dport, $notsent, $tp->snd_cwnd);
    }
}

Running the Trace and Diagnosing Queue Latency

Execute the bpftrace probe on your Linux edge proxy:

sudo bpftrace trace_notsent.bt

When high-volume writes overwhelm socket buffers without low-water mark limits, bpftrace outputs structured alerts:

04:22:15 BUFFERBLOAT ALERT | 10.0.1.15 -> 198.51.100.22:443 | Unsent Bytes: 131072 | CWND: 64
04:22:16 BUFFERBLOAT ALERT | 10.0.1.15 -> 198.51.100.22:443 | Unsent Bytes: 262144 | CWND: 64

This reveals over 256KB of unsent data trapped in the kernel queue, creating significant queuing delay for any concurrent multiplexed streams.

Kernel Tuning and Optimization

To eliminate unsent queue bufferbloat across your fleet:

1. Set a global kernel default for unsent buffer limits via sysctl:

   sudo sysctl -w net.ipv4.tcp_notsent_lowat=16384

   Setting this to 16KB ensures that web servers like Nginx, Envoy, or Node.js only buffer one or two packets of unsent data at a time.
2. Enable tcp_notsent_lowat in Nginx configurations:

   http {
       tcp_notsent_lowat 16384;
   }

3. Verify that priority frames in HTTP/2 multiplexed streams bypass bulk transfer queues cleanly.