How to Trace TCP State Machine Changes and Socket Latency in Linux Using tcpstates

In high-performance Linux network engineering, troubleshooting elusive socket hangs, sudden connection drops, or abnormal connection terminations often requires observing state transitions in real time. Standard command-line tools like netstat and ss provide periodic point-in-time snapshots of sockets, but fail to capture short-lived state changes or measure exactly how many milliseconds a socket spent in an intermediate state (such as SYN_SENT, CLOSE_WAIT, or FIN_WAIT_1). To solve this visibility gap, the BPF Compiler Collection (BCC) provides tcpstates—an eBPF-powered tracing tool that logs every TCP state machine transition system-wide with microsecond precision and negligible kernel overhead.

How tcpstates Uses eBPF to Monitor State Transitions

The Linux TCP/IP stack operates as a deterministic finite-state machine governed by RFC 793 and modern TCP specifications. Whenever a kernel socket transitions between operational states (e.g., from ESTABLISHED to FIN_WAIT_1 or CLOSE_WAIT), the kernel calls internal helper functions such as tcp_set_state().

The tcpstates tool hooks directly into this transition point using kernel probes (kprobes) and tracepoints:

• Timestamp Recording: When a socket enters a new state, tcpstates records a high-resolution timestamp in an internal eBPF hash map indexed by the socket’s memory address.
• Delta Calculation: When the socket transitions to the subsequent state, the tool calculates the exact time spent in the preceding state (in milliseconds) and logs the event to userspace.
• Map Eviction: Once a socket reaches TCP_CLOSE, the corresponding memory reference is purged from the tracking map, avoiding memory leaks.

Installing tcpstates in Linux

The tcpstates utility is bundled with BCC tracing tools. On Ubuntu and Debian-based systems, install the toolset using apt:

sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

On RHEL, AlmaLinux, Rocky Linux, or Fedora:

sudo dnf install bcc-tools kernel-devel-$(uname -r)

Following installation, the executable is available in /usr/sbin/tcpstates-bpfcc or via /usr/share/bcc/tools/tcpstates.

Tracing Active TCP State Transitions

Launch tcpstates with root privileges to begin live system-wide monitoring:

sudo /usr/sbin/tcpstates-bpfcc

As network traffic moves through the system, the terminal outputs real-time transition records:

SKADDR           C-PID C-COMM     LADDR           LPORT RADDR           RPORT OLDSTATE    -> NEWSTATE    MS
ffff9d48b1021400 1842  curl       192.168.1.10    43210 93.184.216.34   443   CLOSE       -> SYN_SENT    0.000
ffff9d48b1021400 0     swapper/2  192.168.1.10    43210 93.184.216.34   443   SYN_SENT    -> ESTABLISHED 18.421
ffff9d48b1021400 1842  curl       192.168.1.10    43210 93.184.216.34   443   ESTABLISHED -> FIN_WAIT1   112.504
ffff9d48b1021400 0     swapper/2  192.168.1.10    43210 93.184.216.34   443   FIN_WAIT1   -> FIN_WAIT2   18.102
ffff9d48b1021400 0     swapper/2  192.168.1.10    43210 93.184.216.34   443   FIN_WAIT2   -> TIME_WAIT   0.210

Key output columns provide essential diagnostic indicators:

• SKADDR: The kernel struct sock memory address, enabling you to trace the complete lifecycle of a single socket across multiple state lines.
• C-PID / C-COMM: Process ID and name responsible for triggering the transition. (Transitions handled in interrupt context by kernel workers appear under swapper).
• OLDSTATE -> NEWSTATE: The previous state and destination state.
• MS: The precise duration spent in the OLDSTATE before the transition occurred.

Filtering by Port and Process

In high-throughput environments, filtering allows you to isolate specific workloads without terminal flooding:

1. Filter by Local Port (e.g., Ingress HTTP/HTTPS)

To inspect state transitions on your local Nginx or Apache server on port 443:

sudo /usr/sbin/tcpstates-bpfcc -L 443

2. Filter by Remote Port (e.g., Egress Database Queries)

To audit outbound database connections connecting to a PostgreSQL server on port 5432:

sudo /usr/sbin/tcpstates-bpfcc -R 5432

3. Filter by Specific Process ID

To monitor socket state handling within a specific backend daemon (e.g., PID 2410):

sudo /usr/sbin/tcpstates-bpfcc -p 2410

4. Include Absolute Timestamps

To synchronize state logs with application trace logs, append the timestamp flag (-T):

sudo /usr/sbin/tcpstates-bpfcc -T

Diagnosing Network Bottlenecks with tcpstates

Examining transition timings allows systems engineers to quickly localize network issues:

• SYN-ACK Latency (SYN_SENT -> ESTABLISHED): Represents the full network Round Trip Time (RTT) of the initial three-way handshake. Significant spikes here point to WAN routing congestion or intermediate firewall inspection delays.
• CLOSE_WAIT Accumulation: If sockets linger in CLOSE_WAIT for thousands of milliseconds, the remote peer has sent a FIN packet, but the local application has failed to call close(), identifying an application socket leak.
• Abrupt Reset Transitions (ESTABLISHED -> CLOSE): Sockets transitioning directly to CLOSE without traversing FIN_WAIT indicate TCP RST packet reception, pointing to peer application crashes, aggressive load-balancer timeouts, or stateful firewall connection drops.

By leveraging tcpstates, Linux engineers obtain granular, kernel-level visibility into TCP socket lifecycles, isolating application-level connection leaks and network handshake latencies with minimal operational impact.