How to Trace TCP Socket TCP_NOTSENT_LOWAT Buffer Bloat in Linux Using eBPF

In high-throughput HTTP/2 and HTTP/3 web servers, multiplexing hundreds of concurrent logical streams over a single TCP connection requires responsive stream prioritisation. If an application fills the socket write queue with megabytes of unsent bulk data (such as a large file download), urgent high-priority frames (such as CSS, critical JavaScript, or API RPCs) are trapped behind queued bytes inside the kernel TCP send buffer. The TCP_NOTSENT_LOWAT socket option mitigates this bufferbloat by signalling pollers (epoll) to wake only when unsent bytes drop below a strict threshold. This tutorial covers tracing TCP_NOTSENT_LOWAT dynamics in Linux using eBPF.

Kernel Mechanics of TCP_NOTSENT_LOWAT

When an application configures setsockopt(fd, IPPROTO_TCP, TCP_NOTSENT_LOWAT, &limit_bytes, sizeof(limit_bytes)), the Linux TCP stack sets tp->notsent_lowat. In tcp_stream_memory_free(), the kernel evaluates whether the unsent data in the write queue (sk->sk_wmem_queued - tp->packets_out * mss) exceeds the threshold. If unsent data exceeds notsent_lowat, EPOLLOUT notifications are withheld, forcing the application event loop to pause buffering and prioritize active multiplexed streams. Tracing this queue reveals write-side bufferbloat latency.

Prerequisites and Environment Verification

Install the required tracing tools on your Linux machine:

sudo apt-get update
sudo apt-get install -y bpftrace linux-headers-$(uname -r)

Verify system sysctl defaults:

sysctl net.ipv4.tcp_notsent_lowat

Authoring the bpftrace TCP_NOTSENT_LOWAT Monitor

Save the following script as trace_notsent.bt to capture write buffer occupancy and poll notification events:

#!/usr/bin/env bpftrace

#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_poll
{
    $sk = (struct sock *)arg1;
    $tp = (struct tcp_sock *)arg1;

    if ($tp->notsent_lowat > 0) {
        $queued = $sk->sk_wmem_queued;
        $unsent = $queued - ($tp->packets_out * 1448);
        @unsent_bytes_dist = hist($unsent);

        if ($unsent > $tp->notsent_lowat) {
            @epoll_blocked_count = count();
        }
    }
}

kprobe:tcp_stream_memory_free
{
    $sk = (struct sock *)arg0;
    $tp = (struct tcp_sock *)arg0;

    if ($tp->notsent_lowat > 0) {
        @wmem_free_checks = count();
    }
}

END
{
    printf("\n--- Unsent Queue Bytes Distribution (Bytes) ---\n");
    print(@unsent_bytes_dist);
    printf("\n--- EPOLLOUT Suppressions Due to Queue Saturation ---\n");
    print(@epoll_blocked_count);
}

Executing the Trace and Analysing Buffer Latency

Run the monitor with root permissions:

sudo bpftrace trace_notsent.bt

Benchmark your HTTP/2 web server using tools like h2load. When TCP_NOTSENT_LOWAT is tuned (e.g. to 16384 bytes), the log2 histogram shows unsent queue sizes tightly clustered around 16 KB instead of ballooning to megabytes:

--- Unsent Queue Bytes Distribution (Bytes) ---
[8K, 16K)           14502 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[16K, 32K)           1204 |@@@@                                              |
[32K, 64K)             42 |                                                  |

This confirms that send-buffer latency is constrained, allowing time-sensitive multiplexed streams to interleave with microsecond precision.

Recommended Web Server Tuning

• For NGINX environments serving HTTP/2, set ssl_buffer_size 4k; alongside tcp_notsent_lowat 16384; in your http context.
• To enforce globally across all outbound sockets without modifying application binaries, configure sysctl -w net.ipv4.tcp_notsent_lowat=16384.