How to Trace TCP QuickACK and Delayed ACK Stalls in Linux Using eBPF

Modern distributed architectures rely on low-latency inter-service RPC communication. However, the classical interplay between Nagle’s algorithm on senders and Delayed Acknowledgments (Delayed ACK) on receivers frequently introduces artificial latency penalties of 40 to 200 milliseconds. When the Linux networking subsystem fails to trigger QuickACK mode, microservice requests experience intermittent response stalls. Tracing these kernel ACK decisions using eBPF exposes exactly when and why delayed acknowledgments occur.

Understanding Linux Kernel Delayed ACK vs QuickACK Mechanics

The Linux TCP stack balances network efficiency against latency through adaptive acknowledgment heuristics:

• Delayed ACK Timer: By default, RFC 1122 permits Linux to delay transmitting an ACK packet for up to TCP_DELACK_MAX (typically 40ms to 200ms) in anticipation of piggybacking the ACK onto outbound response data.
• QuickACK Dynamic Activation: When an application sets the socket option TCP_QUICKACK or when the kernel detects interactive request-response exchanges inside tcp_in_quickack_mode(), it immediately sends pure ACK packets.
• Timer Expiry Latency: If QuickACK mode expires or sender writes trigger Nagle buffering, transactions stall until the kernel timer tcp_delack_timer() fires.

Tracing Delayed ACK Timers in Real Time with bpftrace

Using bpftrace, monitor when the kernel falls back to timer-driven delayed acknowledgments:

sudo bpftrace -e '
#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_delack_timer {
    $sk = (struct sock *)arg0;
    $inet = (struct inet_sock *)arg0;
    $daddr = ntop($inet->inet_daddr);
    $dport = bswap($inet->inet_dport);

    printf("Delayed ACK Timer Fired: target=%s:%d comm=%s pid=%d\n", 
           $daddr, $dport, comm, pid);
    @delack_by_dest[$daddr] = count();
}'

This script logs every occurrence where the kernel waited out the maximum delayed ACK timeout instead of acknowledging incoming payload data immediately.

Measuring Delayed ACK Latency Distribution

To quantify the exact millisecond penalties imposed on client transactions across your infrastructure, record delack completion durations:

sudo bpftrace -e '
kprobe:tcp_send_delayed_ack {
    @ack_dispatches = count();
}

kprobe:tcp_enter_quickack_mode {
    @quickack_transitions = count();
}

interval:s:10 {
    print(@delack_by_dest);
    print(@ack_dispatches);
    print(@quickack_transitions);
}'

Observing high delayed ACK counts alongside low QuickACK transitions confirms that client applications require socket tuning (enabling TCP_NODELAY or programmatic TCP_QUICKACK reactivation) to restore deterministic microsecond latency.