How to Trace TCP QuickACK Delays and Delayed ACK Starvation in Linux Using eBPF

To maximize network efficiency and reduce header overhead, TCP employs Delayed Acknowledgments (RFC 1122), delaying the transmission of ACK packets by up to 200ms in the hope of piggybacking the acknowledgment onto outbound response data. However, in interactive request-response protocols—such as Redis, HTTP microservices, and database RPCs—delayed ACKs collide catastrophically with the sender’s Nagle algorithm. When a sender waits for an ACK before sending subsequent data and the receiver waits for data before sending an ACK, a deadlocked delayed ACK stall occurs. Tracing QuickACK dynamics with eBPF exposes these stalls in production.

Understanding QuickACK Mode vs Delayed ACK Timers

The Linux TCP implementation alternates dynamically between two acknowledgment strategies:

• QuickACK Mode: The socket transmits immediate acknowledgments for every incoming segment (one ACK per packet). Critical during initial connection handshakes and interactive transaction bursts.
• Delayed ACK Mode: The socket arms an acknowledgment timeout timer (ATO), delaying transmission for up to 40ms–200ms unless a second full-sized segment arrives (generating one ACK per two packets).
• QuickACK Exhaustion (icsk_ack.quick): The kernel tracks an internal counter of permitted immediate ACKs. Once this budget decrements to zero, the socket drops out of QuickACK mode and reverts to delayed ACKs.
• TCP_QUICKACK Socket Option: Applications can explicitly force immediate acknowledgments using setsockopt(..., IPPROTO_TCP, TCP_QUICKACK, 1).

Auditing Delayed ACK Counters via netstat

Inspect system-wide delayed ACK frequency and timer expirations before instrumenting kernel probes:

# Check system-wide delayed ACK statistics
netstat -s | grep -i -E "delayed ack"

# Inspect socket internal ACK timers with ss
ss -nti '( dport = :6379 or sport = :6379 )'

The counter delayed acks sent and delayed acks inc reveals how aggressively the kernel delays acknowledgment packets.

Tracing tcp_enter_quickack_mode and Timer Expirations with bpftrace

Attach eBPF probes to tcp_enter_quickack_mode and tcp_delack_timer to detect when sockets transition into immediate acknowledgment mode and when delayed ACK timers expire:

sudo bpftrace -e '
#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_enter_quickack_mode
{
    $sk = (struct sock *)arg0;
    $inet = (struct inet_sock *)arg0;
    $sport = ntohs($inet->inet_sport);
    $dport = ntohs($sk->sk_dport);

    printf("QuickACK Mode Entered: Comm %-12s PID %-6d %d -> %d\n",
           comm, pid, $sport, $dport);
    @quickack_entries = count();
}

kprobe:tcp_delack_timer
{
    $sk = (struct sock *)arg0;
    $tp = (struct tcp_sock *)arg0;

    $ato = $tp->icsk_ack.ato / 1000; // Convert ATO to milliseconds
    printf("Delayed ACK Timer Expired: Comm %-12s PID %-6d Timeout: %u ms\n",
           comm, pid, $ato);
    @delack_expired = count();
}'

If delayed ACK timer expirations match observed application latency spikes (e.g., consistent 40ms delays), your endpoints are suffering from delayed ACK starvation.

Quantifying Inter-ACK Delay Distributions

Measure the exact elapsed duration between receiving an inbound segment and returning the corresponding acknowledgment:

sudo bpftrace -e '
#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_data_queue
{
    $sk = (struct sock *)arg0;
    @packet_rx_time[$sk] = nsecs;
}

kprobe:tcp_send_ack
{
    $sk = (struct sock *)arg0;
    if (@packet_rx_time[$sk]) {
        $delta_us = (nsecs - @packet_rx_time[$sk]) / 1000;
        @ack_latency_us = hist($delta_us);
        delete(@packet_rx_time[$sk]);
    }
}

END
{
    printf("\nInter-ACK Latency Histogram (microseconds):\n");
    print(@ack_latency_us);
    clear(@packet_rx_time);
}'

Eliminating Delayed ACK Stalls in Production Workloads

Apply targeted software and system optimizations based on trace findings:

• Disable Nagle on the Sender (TCP_NODELAY): Set TCP_NODELAY on client and server sockets to ensure small payloads transmit immediately without waiting for previous ACKs.
• Re-enable QuickACK in Application Loops: Because Linux resets QuickACK mode after processing inbound packets, interactive servers should reissue setsockopt(fd, IPPROTO_TCP, TCP_QUICKACK, &one, sizeof(one)) in their primary receive loop.
• Adjust QuickACK Route Metrics: Configure route tables with the quickack 1 attribute for latency-sensitive internal networks:

  sudo ip route change default ... quickack 1