How to Trace TCP Socket TCP_FASTOPEN_CONNECT in Linux Using eBPF

TCP Fast Open (TFO / RFC 7413) eliminates an entire round-trip time (RTT) from TCP handshakes by transmitting data directly inside the initial SYN segment. Historically, client-side applications had to replace the standard connect() system call with sendto(..., MSG_FASTOPEN). To streamline client integration, the Linux kernel introduced the TCP_FASTOPEN_CONNECT socket option via setsockopt(), allowing conventional client libraries to leverage TFO transparently with standard connect() and write() semantics. Tracing TFO connection dynamics with eBPF gives systems architects deep insight into cookie negotiations and SYN retransmission fallback paths.

Mechanics of Client-Side Fast Open Connections

When an application configures TCP_FASTOPEN_CONNECT on an active socket before calling connect(), the kernel intercepts the connect invocation:

1. Rather than transmitting an immediate SYN segment, the kernel defers handshake initiation until the application writes its first data payload (e.g. via write() or send()).
2. Upon receiving the initial data payload, the kernel checks its internal cookie cache for the destination IP.
3. If a valid TFO cookie exists, the kernel encapsulates the data payload directly inside the SYN packet.
4. If no cookie exists, the kernel transmits a SYN requesting a new cookie from the server.

If middleboxes drop SYN packets containing data payloads, the kernel must fall back cleanly to a conventional three-way handshake without userland connection abortion.

Authoring a Custom bpftrace TFO Telemetry Script

To inspect TCP Fast Open connection attempts and trace whether data payloads are successfully attached to SYN segments, write a bpftrace instrumentation script:

#!/usr/bin/env bpftrace

#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_connect
{
  $sk = (struct sock *)arg0;
  $tp = (struct tcp_sock *)arg0;

  if ($tp->fastopen_connect) {
    @tfo_client_attempts = count();
    time("%H:%M:%S: TCP_FASTOPEN_CONNECT on sock %p (cookie_len=%d)\n",
         $sk, $tp->fastopen_req ? $tp->fastopen_req->cookie.len : 0);
  }
}

kprobe:tcp_fastopen_cookie_gen
{
  @tfo_cookies_generated = count();
}

interval:s:10
{
  print(@tfo_client_attempts);
  print(@tfo_cookies_generated);
  clear(@tfo_client_attempts);
  clear(@tfo_cookies_generated);
}

Save this script as trace_tfo_connect.bt. It targets tcp_connect() and inspects the internal fastopen_connect flag and cookie length.

Executing and Evaluating Kernel TFO Tracing

Execute the bpftrace telemetry tool with administrative privileges:

sudo bpftrace trace_tfo_connect.bt

Generate outbound HTTP/HTTPS requests from client applications configured with TCP_FASTOPEN_CONNECT. Review the output metrics:

• First connection attempt: The initial connection to a remote IP displays cookie_len=0, indicating a SYN cookie acquisition probe.
• Subsequent connections: Subsequent handshakes display cookie_len=16 (or matching cipher length), verifying that data payloads are bundled directly inside the SYN packet.
• Fallback monitoring: Cross-reference with nstat -z | grep -i fastopen to ensure TCPSynRetrans counters do not surge, which would indicate middlebox SYN-data drops.

Configuring Host Kernel Fast Open Support

To enable TCP Fast Open client and server modes on Linux servers:

1. Inspect sysctl net.ipv4.tcp_fastopen. A value of 1 enables client-only mode, 2 enables server-only mode, and 3 enables both client and server modes.
2. Set sysctl -w net.ipv4.tcp_fastopen=3 to ensure complete bidirectional support across egress and ingress network paths.