How to Trace TCP Keepalive Probe Dropouts in Linux Using eBPF

Long-lived idle TCP connections—such as database connection pools, gRPC streaming channels, and WebSocket subscriptions—are vulnerable to silent disconnections caused by stateful intermediate firewalls and NAT middleboxes. When state tables expire idle mappings, subsequent packets trigger abrupt connection resets (RST). TCP keepalive probes solve this by periodically transmitting empty ACK packets to refresh NAT state tables and verify peer reachability. Tracing keepalive probe intervals with eBPF exposes premature connection dropouts before applications crash.

Understanding Linux Kernel TCP Keepalive Architecture

The Linux kernel manages TCP keepalive through three coordinated sysctl parameters:

• Keepalive Time (tcp_keepalive_time): The duration of complete connection inactivity before the kernel dispatches the initial keepalive probe (default: 7200 seconds / 2 hours). Modern microservice architectures typically lower this to 300 seconds.
• Keepalive Interval (tcp_keepalive_intvl): The time interval between unacknowledged consecutive probes (default: 75 seconds).
• Keepalive Probes (tcp_keepalive_probes): The maximum number of failed probes allowed before the kernel aborts the connection with an ETIMEDOUT error (default: 9).
• Probe Mechanics: Keepalive probes are generated by sending a 1-byte ACK packet with a sequence number set to snd_una - 1, eliciting an immediate ACK response from healthy peers.

Auditing Active Keepalive Configurations

Inspect existing system parameters and active sockets with keepalive enabled:

# Check global system keepalive defaults (in seconds)
sysctl net.ipv4.tcp_keepalive_time net.ipv4.tcp_keepalive_intvl net.ipv4.tcp_keepalive_probes

# Display active sockets with keepalive timer telemetry
ss -nto 'timer = (keepalive)'

The output column timer:(keepalive,<time_remaining>,<probes_sent>) reveals real-time timer countdowns on active idle flows.

Tracing tcp_keepalive_timer Expirations Using bpftrace

Attach an eBPF kprobe to tcp_keepalive_timer to detect when idle connections fire probes and capture peer addresses:

sudo bpftrace -e '
#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_keepalive_timer
{
    $sk = (struct sock *)arg0;
    $inet = (struct inet_sock *)arg0;
    $tp = (struct tcp_sock *)arg0;

    $sport = ntohs($inet->inet_sport);
    $dport = ntohs($sk->sk_dport);
    $probes_out = $tp->kstat.probes_out;

    if ($probes_out > 0) {
        printf("Keepalive Probe Timeout: PID %-6d Comm %-12s Port %d -> %d (Probes Sent: %u)\n",
               pid, comm, $sport, $dport, $probes_out);
        @failed_keepalive_endpoints[$dport] = count();
    }
}'

This trace alerts engineers whenever peers fail to reply to first-round keepalive probes, identifying dead connections before database queries fail.

Tracing Silent Dropouts and Connection Timeouts

When maximum probes are exhausted without peer response, the kernel invokes tcp_done with an error state. Trace these terminations:

sudo bpftrace -e '
#include <net/sock.h>

kprobe:tcp_done
{
    $sk = (struct sock *)arg0;
    $err = $sk->sk_err;

    if ($err == 110) { // ETIMEDOUT = 110
        printf("TCP Connection Dropped (ETIMEDOUT): PID %-6d Comm %-14s\n", pid, comm);
        @keepalive_abort_count = count();
    }
}'

Tuning Keepalive Parameters for Cloud Microservices

Prevent aggressive cloud NAT gateways (such as AWS NAT Gateways with 350-second idle timeouts) from silently dropping idle sessions:

# Lower system defaults to refresh NAT tables every 4 minutes
sudo sysctl -w net.ipv4.tcp_keepalive_time=240
sudo sysctl -w net.ipv4.tcp_keepalive_intvl=30
sudo sysctl -w net.ipv4.tcp_keepalive_probes=4

Enabling these tuned parameters ensures stateful firewalls retain NAT translations, preventing connection drops during silent periods.