How to Trace TCP Socket TCP_REPAIR_QUEUE Packet Ingestion in Linux Using eBPF

Stateful container migration systems—such as CRIU (Checkpoint/Restore in Userspace)—rely on Linux TCP repair mode to pause, extract, and re-inject in-flight network packets into active socket queues without triggering connection teardowns. The foundational socket option for queue restoration is TCP_REPAIR_QUEUE, which selects which internal queue (receive queue, send queue, or out-of-order queue) is active for userspace inspection and injection. Tracing socket queue manipulation and packet insertion using eBPF provides kernel-level visibility into container live migration fidelity.

Mechanics of TCP Queue Restoration

When an application puts a socket into repair mode, it invokes setsockopt(..., TCP_REPAIR_QUEUE, &queue_id, ...) to select a target queue:

• TCP_RECV_QUEUE (0): The incoming receive buffer holding unacknowledged or unread packets.
• TCP_SEND_QUEUE (1): The outbound send buffer holding unacknowledged in-flight packets.
• TCP_NO_QUEUE (2): Resets active queue selection.

Once a queue is selected, the application injects serialized packet payloads using standard send() calls. The kernel inserts these segments directly into the socket buffer without transmitting packets over the physical network interface. If queue ordering is reversed or sequence numbers are desynchronized, connections crash upon repair mode deactivation.

Authoring a Custom bpftrace Queue Telemetry Script

To inspect when TCP_REPAIR_QUEUE is invoked and monitor packet queue lengths during restoration, write an eBPF script using bpftrace:

#!/usr/bin/env bpftrace

#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_repair_set_queue
{
  $sk = (struct sock *)arg0;
  $queue = arg1;

  time("%H:%M:%S: TCP_REPAIR_QUEUE selected on sock %p (queue=%d)\n", $sk, $queue);
  @queue_selections[$queue] = count();
}

kprobe:tcp_sendmsg_locked
{
  $sk = (struct sock *)arg0;
  $tp = (struct tcp_sock *)arg0;

  if ($tp->repair) {
    @injected_repair_bytes = hist(arg2);
    @repair_injections = count();
  }
}

interval:s:10
{
  print(@queue_selections);
  print(@repair_injections);
  print(@injected_repair_bytes);
  clear(@queue_selections);
  clear(@repair_injections);
  clear(@injected_repair_bytes);
}

Save this script as trace_repair_queue.bt. It instruments tcp_repair_set_queue() and tracks packet injection during active repair mode.

Executing and Evaluating Kernel Telemetry

Run the bpftrace monitoring script with administrative privileges:

sudo bpftrace trace_repair_queue.bt

Trigger a live container checkpoint and restore cycle. Review the output metrics:

• Queue sequencing: Verify that TCP_RECV_QUEUE (0) and TCP_SEND_QUEUE (1) are selected in strict order.
• Injected bytes distribution: Ensure that @injected_repair_bytes matches the unread buffer sizes dumped during the checkpoint phase.
• Sequence integrity: Ensure sequence counters (tp->write_seq and tp->rcv_nxt) match pre-migration states before disabling repair mode.

Host Operational Prerequisites

To safely execute TCP repair queue restoration in production environments:

1. Ensure migration daemons possess CAP_NET_ADMIN capabilities.
2. Disable connection resets: Ensure iptables or nftables rules suppress stray RST segments during IP takeover.
3. Verify ARP/NDP table updates on the target host to redirect physical network traffic seamlessly.