How to Trace TCP State Transitions and Socket Lifecycles Using tcpstates in Linux

Troubleshooting connection resets, lingering half-closed sockets, and socket exhaustion (such as excessive TIME_WAIT states) requires visibility into how TCP sockets progress through the standard RFC 793 state machine. While static commands like netstat or ss provide instantaneous snapshots of socket counts, they cannot measure how long individual sockets remain in each state or pinpoint which application events cause abrupt transitions. The tcpstates utility from the BCC toolkit traces TCP state transitions in real time using eBPF.

Understanding the TCP State Lifecycle

A standard TCP connection transitions through defined states during its lifecycle:

• Connection Handshake: Transitions from CLOSED to SYN_SENT (client) or SYN_RECV (server), terminating in ESTABLISHED.
• Data Transfer: Sockets remain in ESTABLISHED during active payload transmission.
• Connection Teardown: Transitions through FIN_WAIT_1, FIN_WAIT_2, CLOSE_WAIT, LAST_ACK, and TIME_WAIT before finally returning to CLOSED.

Installing the BCC Toolkit

Ensure that BCC tools and kernel development headers are installed on your system:

# On Debian, Ubuntu, or Pop!_OS:
sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

# On RHEL, AlmaLinux, Rocky Linux, or Fedora:
sudo dnf install bcc-tools kernel-devel-$(uname -r)

On Debian/Ubuntu distributions, run the command as tcpstates-bpfcc; on enterprise RHEL distributions, it is located at /usr/share/bcc/tools/tcpstates.

Running tcpstates to Monitor Socket Transitions

To trace TCP state changes system-wide in real time, run tcpstates with root privileges:

sudo tcpstates-bpfcc -T

The -T flag prepends standard human-readable timestamps to every event line.

Example terminal output:

TIME(s)  PID    COMM       IP SADDR:SPORT         DADDR:DPORT         OLDSTATE     -> NEWSTATE    MS
14:10:01 1842   curl       4  192.168.1.50:51230  104.21.5.12:443     SYN_SENT     -> ESTABLISHED 18.24
14:10:01 1842   curl       4  192.168.1.50:51230  104.21.5.12:443     ESTABLISHED  -> FIN_WAIT_1  45.10
14:10:01 1842   curl       4  192.168.1.50:51230  104.21.5.12:443     FIN_WAIT_1   -> FIN_WAIT_2  1.12
14:10:01 1842   curl       4  192.168.1.50:51230  104.21.5.12:443     FIN_WAIT_2   -> TIME_WAIT   0.85
14:10:02 2104   nginx      4  192.168.1.50:80     192.168.1.200:44120 CLOSE_WAIT   -> LAST_ACK    2.40

Filtering by Target Port and Process ID

To eliminate background noise when debugging a specific web server, database, or API service:

# Monitor state transitions exclusively for local port 443:
sudo tcpstates-bpfcc -L 443

# Target a specific backend worker process (PID 1842):
sudo tcpstates-bpfcc -p 1842

Diagnosing Socket Leaks and Teardown Anomalies

By reviewing the OLDSTATE -> NEWSTATE and elapsed time (MS) columns, engineers can diagnose common networking anomalies:

• Lingering CLOSE_WAIT States: Sockets stuck in CLOSE_WAIT indicate that the remote peer initiated connection closure (sent FIN), but the local application process failed to call close() on the file descriptor. This is an application-level socket leak.
• Premature Resets (RST): Direct transitions from ESTABLISHED to CLOSED without traversing FIN_WAIT indicate ungraceful connection termination, typically caused by application crashes, out-of-memory kills, or aggressive network firewall timeouts.
• Extended TIME_WAIT Accumulation: High volumes of sockets lingering in TIME_WAIT for 60 seconds indicate high churn of non-persistent connections; mitigate this by enabling HTTP keep-alives or tuning net.ipv4.tcp_tw_reuse.

Using tcpstates equips administrators with granular, real-time socket lifecycle telemetry to optimize networking performance and resolve elusive connection leaks on Linux servers.