How to Trace TCP Socket TCP_INFO tcpi_retrans_synack TCP TCP-AO Asymmetric Segment Hardware Key Rollover Window Clamp ESP-in-TCP Encap Drops in Linux Using eBPF

Enterprise data centre fabrics and secure hybrid cloud tunnels increasingly rely on RFC 5925 TCP Authentication Option (TCP-AO) to protect long-lived BGP sessions and high-throughput overlay transport links. When combining TCP-AO with IPsec ESP encapsulation over TCP transport (RFC 8229 ESP-in-TCP encapsulation), subtle kernel-level packet drops can emerge during asymmetric hardware key rollovers and window clamping events. Under bursty connection handshakes, failed key derivations or misaligned Master Key Tuples (MKTs) manifest as unexpected SYN-ACK retransmissions tracked in the socket’s tcp_info.tcpi_retrans_synack metric. Tracing these encapsulated packet drops requires low-overhead eBPF kprobes and tracepoints attached to kernel networking hooks. This technical guide outlines how to trace, filter, and diagnose TCP-AO ESP-in-TCP encapsulation drops using eBPF in Linux.

Linux Kernel ESP-in-TCP and TCP-AO Interaction

In the Linux kernel network stack, ESP-in-TCP decapsulation interacts directly with net/ipv4/esp4.c, net/xfrm/xfrm_input.c, and the TCP-AO verification engine in net/ipv4/tcp_ao.c. When an encrypted ESP payload is encapsulated within a TCP framing stream, each segment must satisfy two verification layers: first, the TCP-AO Message Authentication Code (MAC) verifying segment authenticity, and second, the IPsec Security Association (SA) decryption pipeline.

During an automated key rollover, if the transmitting peer increments its Send Key ID (KeyID) before the receiving host completes its local MKT rotation, the TCP-AO verification routine rejects the segment. If this occurs on a connection attempt, the kernel increments tcpi_retrans_synack and invokes kfree_skb() with the reason SKB_DROP_REASON_TCP_AOFAILURE. Without eBPF tracing, identifying whether the failure originated from TCP framing, window clamping, or ESP crypto negotiation is virtually impossible.

Developing the eBPF Tracing Program

We implement a targeted BPF CO-RE (Compile Once – Run Everywhere) program using kprobe:tcp_ao_verify_hash and kprobe:xfrm_input to monitor segment integrity:

#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>

struct drop_event_t {
    u32 saddr;
    u32 daddr;
    u16 sport;
    u16 dport;
    u32 retrans_synack;
    u32 window_clamp;
    u8 keyid_snd;
    u8 keyid_rcv;
    u32 drop_reason;
};

struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} drop_events SEC(".maps");

SEC("kprobe/tcp_ao_verify_hash")
int BPF_KPROBE(trace_tcp_ao_verify, struct sock *sk, struct sk_buff *skb,
               unsigned short int aoh_offset, struct tcp_ao_key *key)
{
    struct tcp_sock *tp = (struct tcp_sock *)sk;
    struct drop_event_t *event;

    u32 retrans = BPF_CORE_READ(tp, retrans_synack);
    u32 clamp = BPF_CORE_READ(tp, window_clamp);

    if (retrans == 0)
        return 0;

    event = bpf_ringbuf_reserve(&drop_events, sizeof(*event), 0);
    if (!event)
        return 0;

    struct inet_sock *inet = (struct inet_sock *)sk;
    event->saddr = BPF_CORE_READ(inet, inet_saddr);
    event->daddr = BPF_CORE_READ(inet, inet_daddr);
    event->sport = bpf_ntohs(BPF_CORE_READ(inet, inet_sport));
    event->dport = bpf_ntohs(BPF_CORE_READ(sk, __sk_common.skc_dport));
    event->retrans_synack = retrans;
    event->window_clamp = clamp;
    event->keyid_snd = BPF_CORE_READ(key, sndid);
    event->keyid_rcv = BPF_CORE_READ(key, rcvid);
    event->drop_reason = 0xAO01;

    bpf_ringbuf_submit(event, 0);
    return 0;
}

char LICENSE[] SEC("license") = "GPL";

Compiling and Loading with bpftool

Compile the BPF source into an ELF object file using Clang/LLVM and attach the program to the target kernel probe points:

# Compile the eBPF probe with modern CO-RE targets
clang -g -O2 -target bpf -D__TARGET_ARCH_x86 -I/usr/include/x86_64-linux-gnu -c tcp_ao_esp_trace.bpf.c -o tcp_ao_esp_trace.bpf.o

# Load and pin the BPF program into the kernel filesystem
bpftool prog load tcp_ao_esp_trace.bpf.o /sys/fs/bpf/tcp_ao_esp_trace type kprobe

# Attach to kernel function entry point
bpftool prog attach pinned /sys/fs/bpf/tcp_ao_esp_trace kprobe tcp_ao_verify_hash

Analyzing Socket Telemetry and Kernel Drops

Monitor the user-space ring buffer or leverage bpftool prog tracelog to capture live drop events:

• Inspect socket statistics with ss -tiepm '( sport = :179 or dport = :179 )' to verify tcpi_retrans_synack counters and window clamp states.
• Correlate captured KeyIDs against configured MKT parameters via ip link show and ip xfrm state.
• Verify that TCP MSS clamping on the tunnel interface does not exceed the ESP-in-TCP framing overhead, preventing fragmentation drops during cryptographic authentication.