How to Trace TCP Listen Socket Overflows and Queue Drops in Linux Using eBPF

High-volume web servers and database proxies frequently encounter rapid surges in inbound connection attempts. If an application’s accept loop fails to drain incoming handshakes rapidly enough, the kernel’s listen backlog queue saturates. When this threshold is breached, the Linux networking stack silently discards incoming SYN packets or resets established client handshakes (ListenOverflows and ListenDrops). Tracing listen queue overflows with eBPF reveals exact queue saturation bottlenecks in real time.

Understanding Linux Kernel Listen Backlog Mechanics

Inbound TCP connection establishment operates across two distinct queues in the Linux kernel:

• SYN Backlog Queue: Tracks half-open connections undergoing three-way handshakes (SYN-RCVD state). Size is governed by net.ipv4.tcp_max_syn_backlog.
• Listen Accept Queue: Holds fully established connections awaiting user space retrieval via accept(). The queue size is capped by the minimum of the application’s listen(backlog) parameter and /proc/sys/net/core/somaxconn.
• Overflow Penalties: When the accept queue fills, the kernel either drops incoming SYN packets or—if net.ipv4.tcp_abort_on_overflow is enabled—replies with a TCP RST, causing immediate connection refused errors on clients.

Tracing Listen Overflows in Real Time with bpftrace

Attach dynamic tracepoints to the kernel’s listen overflow handler to capture saturated sockets instantly:

sudo bpftrace -e '
#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_v4_syn_recv_sock {
    $sk = (struct sock *)arg0;
    if ($sk->sk_ack_backlog > $sk->sk_max_ack_backlog) {
        $inet = (struct inet_sock *)arg0;
        $port = bswap($inet->inet_sport);
        printf("Listen Overflow Detected: port=%d qlen=%d qmax=%d comm=%s\n",
               $port, $sk->sk_ack_backlog, $sk->sk_max_ack_backlog, comm);
        @overflow_by_port[$port] = count();
    }
}'

This script intercepts inbound connection handshakes and triggers an alert whenever an application’s accept queue length exceeds its configured ceiling.

Monitoring System-Wide Queue Drop Metrics

To inspect overall queue drop trends and evaluate whether kernel parameters require adjustment:

sudo bpftrace -e '
tracepoint:tcp:tcp_listen_overflow {
    @listen_overflows[args->lport] = count();
}

interval:s:10 {
    print(@listen_overflows);
    clear(@listen_overflows);
}'

Detecting sustained listen overflows indicates an urgent need to raise net.core.somaxconn, expand application worker pools, or optimize accept thread event loops to handle high-concurrency traffic bursts.