How to Trace TCP Socket TCP_USER_TIMEOUT Abort Latency in Linux Using eBPF

When high-availability distributed systems, replication clusters, or database client pools encounter silent network partition events (such as ungraceful cable disconnects, cloud virtual router failures, or upstream blackholes), default Linux TCP retransmission behavior can keep stalled connections alive for up to 15 to 20 minutes before aborting. The TCP_USER_TIMEOUT socket option (RFC 5482) specifies the exact maximum time in milliseconds that transmitted data can remain unacknowledged before the kernel forcefully terminates the connection with ETIMEDOUT. This tutorial demonstrates how to trace TCP_USER_TIMEOUT abort latencies in Linux using eBPF.

Kernel Mechanics of TCP_USER_TIMEOUT

When an application configures setsockopt(fd, IPPROTO_TCP, TCP_USER_TIMEOUT, &timeout_ms, sizeof(timeout_ms)), the Linux TCP stack sets tp->user_timeout. During subsequent retransmissions within tcp_retransmit_timer(), the kernel computes retransmits_timed_out(sk, retry_until, syn_set). If elapsed retransmission time exceeds icsk_user_timeout, the kernel bypasses exponential backoff limits and immediately calls tcp_write_err(), severing the dead socket. Tracing this subsystem isolates misconfigured timeout thresholds and connection drops.

Prerequisites and Environment Verification

Install the required tracing toolchain on your Linux host:

sudo apt-get update
sudo apt-get install -y bpftrace linux-headers-$(uname -r)

Verify eBPF kprobe functionality:

sudo bpftrace -e 'kprobe:tcp_retransmit_timer { printf("Retransmit timer ticked\n"); exit(); }'

Authoring the bpftrace TCP_USER_TIMEOUT Monitor

Save the following script as trace_user_timeout.bt to capture timeout aborts and unacknowledged retransmission durations:

#!/usr/bin/env bpftrace

#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_write_err
{
    $sk = (struct sock *)arg0;
    $inet = (struct inet_sock *)arg0;
    $tp = (struct tcp_sock *)arg0;

    $daddr = ntop($inet->sk.__sk_common.skc_daddr);
    $dport = bswap($inet->sk.__sk_common.skc_dport);
    $saddr = ntop($inet->sk.__sk_common.skc_rcv_saddr);
    $sport = bswap($inet->inet_sport);

    if ($sk->sk_err == 110) { // 110 = ETIMEDOUT
        printf("[TCP TIMEOUT ABORT] %s:%d -> %s:%d aborted after user timeout! Configured: %d ms\n",
            $saddr, $sport, $daddr, $dport, $tp->icsk_user_timeout);
        @timeout_aborts[$daddr, $dport] = count();
    }
}

kprobe:tcp_retransmit_skb
{
    $sk = (struct sock *)arg0;
    $tp = (struct tcp_sock *)arg0;

    if ($tp->icsk_user_timeout > 0) {
        @tfo_user_timeout_retransmits = count();
    }
}

END
{
    printf("\n--- User Timeout Abort Telemetry ---\n");
    print(@timeout_aborts);
    print(@tfo_user_timeout_retransmits);
}

Running the Trace and Observing Abort Timing

Run the monitor with administrative privileges:

sudo bpftrace trace_user_timeout.bt

Simulate an upstream network blackhole using iptables -I OUTPUT -d <destination> -j DROP. Instead of waiting 15 minutes, sockets configured with a 5000ms user timeout trigger immediate abort notifications:

[TCP TIMEOUT ABORT] 10.0.1.20:41200 -> 10.0.2.80:5432 aborted after user timeout! Configured: 5000 ms
[TCP TIMEOUT ABORT] 10.0.1.20:41204 -> 10.0.2.80:5432 aborted after user timeout! Configured: 5000 ms

Recommended Practices for Resilient Cloud Architecture

• Always pair TCP_USER_TIMEOUT with SO_KEEPALIVE: when a connection is completely idle, keepalive probes induce retransmissions, ensuring TCP_USER_TIMEOUT detects dead connections even when no client requests are actively in flight.
• For internal microservices across reliable cloud data centres, set TCP_USER_TIMEOUT = 10000 (10 seconds) to rapidly evict dead nodes from load-balancer pools.
• For WAN and mobile clients, configure conservative values (e.g. 30,000–60,000 ms) to avoid prematurely severing connections experiencing transient cellular handoffs.