How to Trace TCP Out-of-Memory Socket Pruning in Linux Using eBPF

Under heavy network traffic or severe memory pressure, the Linux kernel can exhaust the memory limits allocated to the networking subsystem (net.ipv4.tcp_mem). When global TCP memory limits are breached, the kernel activates emergency pruning routines (such as tcp_prune_queue and tcp_collapse). During pruning, the kernel discards unacknowledged out-of-order packet queues and collapses fragmented sk_buff buffers to recover kernel pages. Tracing these memory-induced stalls with eBPF reveals hidden transport latency spikes in production web servers.

Understanding Linux TCP Memory Allocation and Pruning Tiers

The Linux TCP stack regulates memory through three distinct threshold parameters defined in net.ipv4.tcp_mem (measured in 4KB memory pages):

• Low Threshold (min): The kernel applies no memory pressure constraints; sockets freely allocate receive and transmit buffers.
• Pressure Threshold (pressure): The kernel enters memory pressure mode. Socket buffer auto-tuning freezes and allocations are constrained.
• High Threshold (max): The absolute ceiling. If reached, the kernel triggers emergency socket pruning:
  • Out-of-Order Queue Discard: Packets received out of order waiting in tp->out_of_order_queue are dropped immediately, forcing the sender to retransmit.
  • Buffer Collapsing: Multiple small socket buffers (skbs) are merged into contiguous pages to eliminate descriptor overhead.
  • Packet Drops: Inbound packets are dropped immediately before entering receive queues.

Auditing Active TCP Memory Footprint

Inspect current kernel memory allocations and pruning counters before instrumenting probes:

# Inspect active TCP memory limits (min, pressure, max in 4KB pages)
sysctl net.ipv4.tcp_mem

# Check active kernel network memory utilization
cat /proc/net/sockstat

# Inspect system-wide pruning and collapse counters
netstat -s | grep -i -E "prune|collapse|memory"

If packets pruned from receive queue or packet headers collapsed is incrementing, socket pruning is actively occurring.

Tracing tcp_prune_queue and tcp_collapse with bpftrace

Attach eBPF probes to catch when emergency pruning routines are invoked and measure which processes are impacted:

sudo bpftrace -e '
#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_prune_queue
{
    $sk = (struct sock *)arg0;
    $tp = (struct tcp_sock *)arg0;

    printf("Emergency TCP Prune: PID %-6d Comm %-14s SkMem: %u bytes\n",
           pid, comm, $sk->sk_forward_alloc);
    @prune_events = count();
}

kprobe:tcp_collapse
{
    $sk = (struct sock *)arg0;
    printf("Buffer Collapse Triggered: PID %-6d Comm %-14s\n", pid, comm);
    @collapse_events = count();
}'

This script alerts systems engineers the exact instant kernel buffers run out of memory space, exposing which application workloads trigger memory pressure.

Measuring Latency Penalties Induced by Buffer Collapsing

Collapsing sk_buff descriptors requires CPU-intensive memory copies that lock the socket, adding milliseconds of thread stalls. Quantify this overhead:

sudo bpftrace -e '
#include <linux/skbuff.h>

kprobe:tcp_collapse
{
    @collapse_start[tid] = nsecs;
}

kretprobe:tcp_collapse
{
    if (@collapse_start[tid]) {
        $delta_us = (nsecs - @collapse_start[tid]) / 1000;
        @collapse_lat_us = hist($delta_us);
        delete(@collapse_start[tid]);
    }
}

END
{
    printf("\nTCP Collapse Latency Histogram (microseconds):\n");
    print(@collapse_lat_us);
    clear(@collapse_start);
}'

Tuning Network Stack Memory Boundaries

Prevent premature socket pruning on high-throughput database and container hosts:

• Increase Global TCP Memory Limits: Scale tcp_mem proportionally with physical system RAM:

  # Scale tcp_mem (example for 32GB RAM host)
  sudo sysctl -w net.ipv4.tcp_mem="786432 1048576 1572864"

• Adjust Socket Receive Max Limits: Ensure individual socket caps in net.ipv4.tcp_rmem do not starve sibling connections when concurrent connections spike.