How to Trace TCP SYN Backlog Latency and Drops in Linux Using bpftrace

In high-performance web servers, reverse proxies, and microservices architectures, unexplained request latency often originates during initial TCP socket handshakes before application workers ever receive the connection. When inbound connection rates spike, the kernel’s SYN backlog (half-open connection queue) can become saturated, resulting in silent SYN drops or prolonged handshake completion times. While tools like netstat report cumulative drop counters, they offer no granular insight into per-socket latency distributions. By deploying lightweight bpftrace one-liners and scripts, administrators can measure SYN backlog latency directly inside the kernel network stack.

The Mechanics of TCP Handshakes and Kernel Queues

When an inbound TCP SYN packet arrives at a listening network interface:

1. The kernel verifies whether the listening socket’s SYN backlog limit (defined by tcp_max_syn_backlog) has room.
2. If space is available, the kernel creates a lightweight request_sock structure, sends a SYN-ACK, and queues the half-open socket into the SYN backlog.
3. Once the remote client replies with the concluding ACK packet, the kernel transitions the connection into a full struct sock and transfers it into the accept queue (listen backlog).

If client ACKs are delayed, intermediate networks experience packet loss, or the server CPU is overloaded with interrupt handling, sockets linger in the SYN backlog. Tracing the time elapsed between tcp_v4_conn_request() and tcp_v4_syn_recv_sock() reveals the exact distribution of SYN handshake latency.

Prerequisites and Tool Installation

To execute bpftrace scripts, your system requires root privileges and a kernel built with BPF Type Format (BTF) support (Linux 4.19+ or Linux 5.4+ recommended):

Ubuntu / Debian

sudo apt update
sudo apt install bpftrace linux-headers-$(uname -r)

RHEL / Rocky Linux / Fedora

sudo dnf install bpftrace bpftrace-tools

Tracing SYN Handshake Latency with a bpftrace One-Liner

To measure the time elapsed from the initial inbound SYN packet until the completed handshake creates a full socket, run the following bpftrace command:

sudo bpftrace -e '
kprobe:tcp_v4_conn_request {
    @start[arg0] = nsecs;
}

kretprobe:tcp_v4_syn_recv_sock {
    $sk = (uint64)retval;
    if ($sk != 0) {
        $req = (uint64)arg0;
        if (@start[$req]) {
            @handshake_latency_us = hist((nsecs - @start[$req]) / 1000);
            delete(@start[$req]);
        }
    }
}

END {
    clear(@start);
}'

When you terminate the script with Ctrl + C, bpftrace renders a power-of-two logarithmic histogram:

@handshake_latency_us: 
[16, 32)               4 |@@@@                                                |
[32, 64)             128 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[64, 128)             42 |@@@@@@@@@@@@@@@@@                                   |
[128, 256)             8 |@@@                                                 |
[256, 512)             1 |                                                    |

This histogram shows that the vast majority of TCP handshakes complete within 32 to 64 microseconds, while outliers taking over 256 microseconds may indicate network retransmissions on the final ACK.

Tracing Dropped SYN Packets in Real Time

When the kernel accept queue overflows or SYN cookies trigger, the kernel drops inbound SYN segments via tcp_v4_conn_request(). You can write a targeted bpftrace probe to record every instance where a connection request fails:

sudo bpftrace -e '
kprobe:tcp_v4_conn_request {
    $sk = (struct sock *)arg0;
    $skb = (struct sk_buff *)arg1;
    // Log listening port and connection attempt
}

kprobe:tcp_listendrop {
    $sk = (struct sock *)arg0;
    printf("TIME: %s | COMM: %s | PID: %d | LISTEN PORT: %d | SYN/LISTEN QUEUE OVERFLOW
",
        time("%H:%M:%S"), comm, pid, $sk->__sk_common.skc_num);
}'

Whenever a socket backlog drop occurs, the console alerts you with the exact timestamp, listening application, and port number.

Kernel Tuning and Remediation Strategies

• Expand SYN Backlog Limit: If SYN queues saturate during traffic spikes, inspect current limits via sysctl net.ipv4.tcp_max_syn_backlog and increase the capacity (e.g., to 8192 or 16384) in /etc/sysctl.d/99-network.conf.
• Enable TCP SYN Cookies: Ensure SYN flood protection is active by verifying net.ipv4.tcp_syncookies = 1, which allows legitimate handshakes to proceed via cryptographic hashes even when backlogs overflow.
• Align Listen Backlog: Check that your application’s listen() backlog parameter matches the kernel’s net.core.somaxconn value to prevent handshakes from completing in the SYN queue only to be dropped at the accept queue stage.