How to Trace TCP Socket TCP_REPAIR State Migration in Linux Using eBPF

In high-availability cloud platforms and container orchestration engines, migrating a live process or container from one physical host to another without severing established TCP connections is accomplished through Checkpoint/Restore in Userspace (CRIU). At the heart of transparent TCP connection migration lies the Linux kernel socket option TCP_REPAIR. By placing a socket into repair mode, user-space utilities can pause network traffic, dump send/receive queues, clone sequence numbers, and reconstruct identical socket state on the destination host without sending reset (RST) packets. This tutorial covers tracing TCP_REPAIR state transitions using eBPF.

The Mechanics of TCP_REPAIR in the Kernel

Enabling setsockopt(fd, IPPROTO_TCP, TCP_REPAIR, &val, sizeof(val)) halts all TCP state machines, timer expirations, and acknowledgment dispatches. While in repair mode:

• TCP_REPAIR_QUEUE selects either the receive queue (TCP_RECV_QUEUE) or send queue (TCP_SEND_QUEUE) for inspection.
• Sequence numbers (tp->write_seq and tp->rcv_nxt) become read-write via TCP_QUEUE_SEQ.
• Socket options, window scaling factors, and timestamps can be restored manually on a newly synthesized socket before leaving repair mode.

Tracing tcp_repair_socket() and sequence number restorations ensures live migration reliability and pinpoints socket recreation latency.

Prerequisites and Toolchain Setup

Install the required tracing tools on your Linux server:

sudo apt-get update
sudo apt-get install -y bpftrace linux-headers-$(uname -r) criu

Confirm kernel support for TCP_REPAIR:

sudo bpftrace -e 'kprobe:tcp_repair_set_options { printf("TCP_REPAIR active\n"); exit(); }'

Authoring the bpftrace TCP_REPAIR Telemetry Script

Save the following script as trace_tcprepair.bt to capture socket repair activations, queue dumps, and migration timings:

#!/usr/bin/env bpftrace

#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_repair_set_options
{
    $sk = (struct sock *)arg0;
    $val = arg1;

    printf("[TCP_REPAIR] Socket %p entering repair mode (state flag: %d)\n", $sk, $val);
    @repair_events[comm] = count();
    @migration_start[tid] = nsecs;
}

kprobe:tcp_repair_socket
{
    $sk = (struct sock *)arg0;
    $val = arg1;

    if ($val == 0) { // Exiting repair mode
        if (@migration_start[tid]) {
            $duration_us = (nsecs - @migration_start[tid]) / 1000;
            printf("[TCP_REPAIR DONE] Socket %p resumed normal TCP processing. Repair window: %d us\n",
                $sk, $duration_us);
            @repair_latency_us = hist($duration_us);
            delete(@migration_start[tid]);
        }
    }
}

END
{
    printf("\n--- TCP_REPAIR Migration Latency Distribution (us) ---\n");
    print(@repair_latency_us);
    print(@repair_events);
}

Running the Trace During Container Checkpoints

Execute the bpftrace script with root privileges:

sudo bpftrace trace_tcprepair.bt

Initiate a CRIU checkpoint or live container migration: criu dump -t <pid> --tcp-established. Output reveals the exact millisecond intervals where socket queues are frozen and serialized:

[TCP_REPAIR] Socket 0xffff8801b7a2a000 entering repair mode (state flag: 1)
[TCP_REPAIR DONE] Socket 0xffff8801b7a2a000 resumed normal TCP processing. Repair window: 1420 us

Sub-millisecond repair windows confirm that live migration pauses network traffic imperceptibly without triggering client-side connection resets.

Production Security and Performance Considerations

• TCP_REPAIR requires CAP_NET_ADMIN privileges to prevent unprivileged users from injecting rogue sequence numbers into existing connections.
• Ensure network switches update ARP tables immediately after migration via gratuitous ARP (GARP) packets to avoid packet routing loops after sockets exit repair mode.