How to Trace TCP Receive Buffer Collapsing and Window Drops in Linux Using eBPF

In high-throughput Linux storage nodes and API gateways, memory constraints can force the kernel networking subsystem to collapse socket receive buffers (tcp_collapse()). When socket memory allocations approach sysctl bounds, the kernel aggressively merges adjacent sk_buff structures and discards out-of-order queues to reclaim slab memory. This buffer collapse triggers TCP window drops and severe latency spikes that standard network interface counters fail to explain.

Understanding Linux Kernel TCP Buffer Collapsing Mechanics

When user space applications read data slower than incoming network line rates, socket memory fills:

• Memory Limits: The kernel enforces per-socket memory thresholds defined in /proc/sys/net/ipv4/tcp_rmem (min, default, max).
• The Collapse Routine: If a socket breaches its memory quota, tcp_try_rmem_schedule() fails, forcing invocation of tcp_collapse(). This routine traverses the receive queue, repacking packet payloads into unified buffers to eliminate memory overhead.
• Window Shrinkage: If collapsing fails to recover sufficient space, the kernel drops newly arrived packets and advertises a zero-window (ZWIN) back to the sender, pausing TCP transfer.

Tracing tcp_collapse Invocations with bpftrace

Attach dynamic instrumentation to tcp_collapse to observe buffer compaction events directly:

sudo bpftrace -e '
#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_collapse {
    $sk = (struct sock *)arg0;
    $inet = (struct inet_sock *)arg0;
    $daddr = ntop($inet->inet_daddr);
    $dport = bswap($inet->inet_dport);

    printf("TCP Collapse: pid=%d comm=%s local_port=%d remote_ip=%s\n", 
           pid, comm, $dport, $daddr);
    @collapses_by_proc[comm] = count();
}'

This script logs every socket buffer collapse alongside the offending process name and client IP address in real time.

Correlating Memory Pressure and Window Retransmissions

To inspect whether collapses correlate with application thread starvation or undersized memory buffers, monitor socket allocation failures:

sudo bpftrace -e '
kprobe:tcp_prune_queue {
    @queue_prunes = count();
}

kprobe:tcp_drop {
    @tcp_drops = count();
}

interval:s:10 {
    print(@collapses_by_proc);
    print(@queue_prunes);
    print(@tcp_drops);
}'

Frequent collapse and prune triggers indicate an urgent need to tune net.ipv4.tcp_rmem or profile application worker thread event loops to prevent socket buffer exhaustion.