How to Trace TCP Connect Latency and Socket Backlog Queues in Linux Using eBPF

High-concurrency network servers often suffer from latency spikes when connection establishment slows down during peak request traffic. While application logs might report elevated response times, diagnosing whether the delay originates in network round-trip time (RTT), kernel TCP three-way handshake processing, or socket listen backlog queue starvation requires low-level kernel visibility. Extended Berkeley Packet Filters (eBPF) enable precise, microsecond-level measurement of the time spent between a client’s SYN packet and the server application’s accept() call.

Understanding Linux Connection Queuing: SYN Queue vs Listen Backlog

The Linux TCP kernel stack implements a two-stage queue architecture to manage incoming connections:

• SYN Queue (Half-Open Sockets): Governed by net.ipv4.tcp_max_syn_backlog. Holds connections in SYN_RECV state waiting for the client’s completing ACK packet.
• Accept Queue / Listen Backlog: Governed by the backlog argument in listen(fd, backlog) clamped by net.core.somaxconn. Holds established connections in TCP_ESTABLISHED state waiting for the userspace application to invoke accept().
• Backlog Overflow: If the userspace application thread is blocked (e.g., event loop stall or thread pool exhaustion), the accept queue fills. Subsequent connection attempts are silently dropped or reset via SYN cookies, causing severe latency spikes.

Tracing TCP Connect Latency with bpftrace

Deploy bpftrace to calculate the duration of the TCP three-way handshake by measuring time between tcp_v4_syn_recv_sock and the userspace socket dequeue:

sudo bpftrace -e '
#include <net/sock.h>
#include <net/tcp.h>

kprobe:tcp_v4_syn_recv_sock
{
    $sk = (struct sock *)arg1;
    @start[arg1] = nsecs;
}

kretprobe:inet_csk_accept
{
    $sk = (struct sock *)retval;
    if ($sk != 0 && @start[(uint64)$sk]) {
        $lat_us = (nsecs - @start[(uint64)$sk]) / 1000;
        @accept_latency_us = hist($lat_us);
        delete(@start[(uint64)$sk]);
    }
}'

When running, bpftrace outputs an intuitive logarithmic histogram showing how long established connections spent waiting in the kernel queue before the application called accept():

@accept_latency_us: 
[16, 32)             45 |@@@@                                    |
[32, 64)            312 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@            |
[64, 128)           420 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[128, 256)           85 |@@@@@@@@                                |
[256, 512)            8 |@                                       |
[512, 1024)           2 |                                        |

Detecting Socket Listen Queue Overflow in Real Time

Use BCC’s tcpdrops or author an eBPF probe monitoring tcp_v4_conn_request to detect when incoming connections are dropped due to full backlogs:

sudo bpftrace -e '
#include <net/sock.h>
#include <net/request_sock.h>

kprobe:tcp_v4_conn_request
{
    $sk = (struct sock *)arg0;
    $inet = (struct inet_sock *)$sk;
    $qlen = $sk->sk_ack_backlog;
    $max_qlen = $sk->sk_max_ack_backlog;
    
    if ($qlen >= $max_qlen) {
        printf("BACKLOG OVERFLOW: Port %d | Queue: %d / Max: %d | Dropping SYN
",
               bswap($inet->inet_sport), $qlen, $max_qlen);
    }
}'

Kernel Tuning and Mitigation Recommendations

When eBPF tracing reveals backlog exhaustion or excessive accept delays:

1. Increase System Maximum Connection Limits:

   sudo sysctl -w net.core.somaxconn=65535
   sudo sysctl -w net.ipv4.tcp_max_syn_backlog=65535

2. Tune Application Backlog Parameter: Ensure your web server (e.g., Nginx listen 443 backlog=65535; or Node.js server.listen(port, host, 65535)) configures an accept queue matching system limits.
3. Enable TCP SO_REUSEPORT: Distribute connection acceptance across multiple CPU worker threads to eliminate single-core accept queue bottlenecks.