How to Trace TCP Socket SO_KEEPALIVE Probing and Idle Timeouts in Linux Using eBPF

Long-lived TCP connections—such as database connection pools, gRPC streams, and reverse proxy reverse tunnels—frequently cross stateful firewalls and NAT gateways. If a connection remains silent beyond intermediary session timeouts, middleboxes drop connection state without transmitting reset (RST) packets. The Linux socket option SO_KEEPALIVE ensures periodic probes detect dead peers and preserve NAT mappings. This guide details how to trace TCP keepalive timer activations and dropped connections in Linux using eBPF.

Linux Kernel TCP Keepalive Architecture

The Linux kernel TCP stack manages keepalive probing via three configurable sysctl parameters and socket options:

• tcp_keepalive_time (default: 7200 seconds): Idle duration before the kernel dispatches the initial keepalive probe packet.
• tcp_keepalive_intvl (default: 75 seconds): Interval between unacknowledged probe retransmissions.
• tcp_keepalive_probes (default: 9): Maximum consecutive unanswered probes before the kernel terminates the connection with ETIMEDOUT.

When an application configures setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, &val, sizeof(val)), the kernel arms tcp_keepalive_timer(). Tracing this timer using eBPF illuminates real-time connection health and silent peer drops.

Prerequisites and Toolchain Setup

Install the required tracing tools on your Linux workstation or server:

sudo apt-get update
sudo apt-get install -y bpftrace linux-headers-$(uname -r) iproute2

Confirm kernel BTF symbol availability for bpftrace:

test -f /sys/kernel/btf/vmlinux && echo "Kernel BTF is active" || echo "BTF missing"

Authoring the bpftrace Keepalive Monitor

Create a bpftrace monitoring script named trace_keepalive.bt to capture keepalive timers and probe dispatches:

#!/usr/bin/env bpftrace

#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_keepalive_timer
{
    $sk = (struct sock *)arg0;
    $inet = (struct inet_sock *)arg0;
    $tp = (struct tcp_sock *)arg0;

    $daddr = ntop($inet->sk.__sk_common.skc_daddr);
    $dport = bswap($inet->sk.__sk_common.skc_dport);
    $saddr = ntop($inet->sk.__sk_common.skc_rcv_saddr);
    $sport = bswap($inet->inet_sport);

    // Filter active keepalive sockets
    if ($tp->keepalive_probes > 0) {
        printf("[KEEPALIVE PROBE] %s:%d -> %s:%d (Unacked Probes: %d / %d)\n",
            $saddr, $sport, $daddr, $dport,
            $tp->keepalive_probes, $tp->keepalive_probes);
        @probe_counts[$daddr, $dport] = count();
    }
}

kprobe:tcp_done
{
    $sk = (struct sock *)arg0;
    $inet = (struct inet_sock *)arg0;
    $tp = (struct tcp_sock *)arg0;

    // Detect if socket closed due to keepalive timeout
    if ($sk->sk_err == 110) { // 110 = ETIMEDOUT
        $daddr = ntop($inet->sk.__sk_common.skc_daddr);
        $dport = bswap($inet->sk.__sk_common.skc_dport);
        printf("[CONNECTION DROPPED] Peer %s:%d terminated by keepalive timeout!\n",
            $daddr, $dport);
        @dropped_peers[$daddr] = count();
    }
}

END
{
    printf("\n--- Keepalive Activity Summary ---\n");
    print(@probe_counts);
    print(@dropped_peers);
}

Running the Probe and Analysing Keepalive Events

Execute the script with root privileges:

sudo bpftrace trace_keepalive.bt

When connections cross idle thresholds or remote endpoints disappear abruptly (such as ungraceful container halts or severed WAN links), output displays live probe activity:

[KEEPALIVE PROBE] 192.168.1.105:44512 -> 10.0.0.24:5432 (Unacked Probes: 1 / 9)
[KEEPALIVE PROBE] 192.168.1.105:44512 -> 10.0.0.24:5432 (Unacked Probes: 2 / 9)
[CONNECTION DROPPED] Peer 10.0.0.24:5432 terminated by keepalive timeout!

Optimising System-Wide Keepalive Thresholds

For cloud-native deployments traversing firewalls with 5-minute idle drops, tune system defaults via /etc/sysctl.d/99-tcp-keepalive.conf:

# Send first probe after 120 seconds of silence
net.ipv4.tcp_keepalive_time = 120

# Resend probe every 15 seconds if unanswered
net.ipv4.tcp_keepalive_intvl = 15

# Drop connection after 4 consecutive unanswered probes (total 180s)
net.ipv4.tcp_keepalive_probes = 4

Apply the settings immediately with sudo sysctl --system to guarantee responsive connection recycling across high-traffic microservices.