How to Trace TCP Socket TCP_DEFER_ACCEPT Handshake Delays in Linux Using eBPF

High-performance web servers such as NGINX and Apache frequently deploy the TCP_DEFER_ACCEPT socket option. Instead of waking the application event loop immediately upon completion of the initial three-way TCP handshake (SYN, SYN-ACK, ACK), the Linux kernel holds the connected socket in the kernel listen backlog until the client transmits its first payload byte (such as an HTTP request line). While this prevents thread wakeups on idle clients, slow or stalled clients can saturate listener backlogs. This tutorial demonstrates how to trace TCP_DEFER_ACCEPT transitions using eBPF on Linux.

The Mechanics of TCP_DEFER_ACCEPT in the Kernel

When an application configures setsockopt(fd, IPPROTO_TCP, TCP_DEFER_ACCEPT, &timeout, sizeof(timeout)), the socket behavior in tcp_check_req() and tcp_v4_syn_recv_sock() changes. The kernel acknowledges the client’s ACK but postpones calling sock_def_readable(). A timer elapses up to timeout seconds. If the client delivers application data before expiration, the connection immediately emerges from accept() ready for reading. If the timer elapses with no data, the kernel drops the connection or retransmits SYN-ACK. Tracing this interval exposes misbehaving clients and backlog saturation.

Prerequisites and Toolchain Setup

Install the necessary tracing dependencies on your Linux machine:

sudo apt-get update
sudo apt-get install -y bpftrace linux-headers-$(uname -r)

Confirm that the system supports eBPF kprobes:

sudo bpftrace -e 'kprobe:tcp_v4_syn_recv_sock { printf("SYN received\n"); exit(); }'

Authoring the bpftrace TCP_DEFER_ACCEPT Monitor

Save the following script as trace_defer_accept.bt to monitor delayed wakeups and identify connection timeout drops:

#!/usr/bin/env bpftrace

#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_check_req
{
    $sk = (struct sock *)arg0;
    $inet = (struct inet_sock *)arg0;
    $daddr = ntop($inet->sk.__sk_common.skc_daddr);
    $dport = bswap($inet->sk.__sk_common.skc_dport);

    @handshake_start[$daddr, $dport] = nsecs;
}

kprobe:tcp_data_ready
{
    $sk = (struct sock *)arg0;
    $inet = (struct inet_sock *)arg0;
    $daddr = ntop($inet->sk.__sk_common.skc_daddr);
    $dport = bswap($inet->sk.__sk_common.skc_dport);

    if (@handshake_start[$daddr, $dport]) {
        $defer_delay_ms = (nsecs - @handshake_start[$daddr, $dport]) / 1000000;
        printf("[DEFER PAYLOAD] Client %s:%d sent first byte after %d ms\n",
            $daddr, $dport, $defer_delay_ms);
        @defer_duration_ms = hist($defer_delay_ms);
        delete(@handshake_start[$daddr, $dport]);
    }
}

END
{
    printf("\n--- TCP_DEFER_ACCEPT Delay Distribution (ms) ---\n");
    print(@defer_duration_ms);
    clear(@handshake_start);
}

Running the Trace and Analysing Delays

Execute the bpftrace monitor with root privileges:

sudo bpftrace trace_defer_accept.bt

Simulate HTTP client traffic or trigger test connections via netcat. Output captures the exact pause between handshake completion and the arrival of the first data byte:

[DEFER PAYLOAD] Client 10.0.0.15:43810 sent first byte after 2 ms
[DEFER PAYLOAD] Client 192.168.1.80:59124 sent first byte after 184 ms

Connections taking hundreds of milliseconds before sending data may indicate slow TLS handshakes, network latency, or slow-loris attacks attempting to hold listener backlog slots open.

Backlog Tuning and Server Configuration

If listener queues saturate under high traffic:

• Tune NGINX configuration: set listen 80 deferred; with conservative timeouts (e.g. 5–10 seconds) rather than large defaults.
• Increase kernel listen backlog queue: sysctl -w net.core.somaxconn=65535.
• Monitor listener overflow drops using netstat -s | grep "listen queue".