How to Trace TCP Connection Lifespans and Throughput in Linux Using tcplife

Diagnosing transient latency spikes, abnormal socket closures, and microburst network throughput in production Linux environments often proves difficult using standard polling utilities like netstat or ss. Because active connections frequently open, exchange brief payloads, and close within milliseconds, snapshot tools miss them entirely. Meanwhile, capturing full packet streams with tcpdump introduces substantial CPU overhead and generates unwieldy trace files. To resolve this dilemma, the BPF Compiler Collection (BCC) provides tcplife—an eBPF-powered tracing tool that monitors kernel TCP socket lifecycle events in real time, reporting duration and throughput summaries upon connection close with negligible overhead.

How tcplife Leverages eBPF for Socket Tracing

Unlike userspace sniffers that copy packets from network drivers, tcplife attaches Extended Berkeley Packet Filters (eBPF) directly to kernel tracepoints and kprobes governing TCP state transitions:

• tcp_set_state(): When a socket transitions to TCP_ESTABLISHED, tcplife records a timestamp and initializes connection tracking in an internal BPF map.
• Data Transfer Accounting: The probe monitors socket counters tracking bytes transmitted (tx_b) and bytes received (rx_b) across the lifetime of the socket.
• tcp_close(): When the socket enters TCP_CLOSE, tcplife calculates the total elapsed duration in milliseconds, emits a structured event line to userspace, and frees the tracking map entry.

Installing tcplife in Linux

The tcplife utility is distributed as part of the BCC tools package. On Ubuntu and Debian-based systems, install the toolset via apt:

sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

On RHEL, Rocky Linux, or Fedora:

sudo dnf install bcc-tools kernel-devel-$(uname -r)

On modern distributions, BCC binaries are installed into /usr/sbin with a -bpfcc suffix (such as tcplife-bpfcc) or accessible via /usr/share/bcc/tools/tcplife.

Tracing TCP Connections with tcplife

Execute tcplife with root privileges to begin live session tracing across all network interfaces:

sudo /usr/sbin/tcplife-bpfcc

When active network sockets establish and terminate, tcplife outputs formatted tabular records:

PID   COMM       LADDR           LPORT RADDR           RPORT TX_KB RX_KB MS
1420  nginx      192.168.1.10    443   10.0.0.45       52314 142   12    84.32
2891  curl       192.168.1.10    48922 93.184.216.34   80    1     48    120.15
3102  postgres   192.168.1.10    5432  192.168.1.25    39810 5     18    12.40

Key columns provide actionable diagnostics:

• PID / COMM: The process identifier and process name that owned the socket.
• LADDR / LPORT: Local IP address and listening/source port.
• RADDR / RPORT: Remote client or server IP address and port.
• TX_KB / RX_KB: Total volume of data transmitted and received in kilobytes.
• MS: The total socket lifespan from establishment to termination in milliseconds.

Advanced Filtering and Filtering by Port

In high-traffic production environments, tracing every ephemeral connection across the system generates excessive terminal noise. You can filter tcplife to focus on specific operational criteria:

1. Trace Connections for a Specific Local Port (e.g., HTTPS Port 443)

To inspect only web server ingress connections and ignore database and management traffic:

sudo /usr/sbin/tcplife-bpfcc -L 443

2. Trace Connections by Remote Port (e.g., PostgreSQL Port 5432)

To audit outbound database connections initiated by application microservices:

sudo /usr/sbin/tcplife-bpfcc -R 5432

3. Filter by Process Identifier (PID)

To isolate connection durations generated by a specific backend daemon (e.g., PID 4120):

sudo /usr/sbin/tcplife-bpfcc -p 4120

4. Include High-Resolution Timestamps

To correlate socket lifespans with application logs or system metric collectors, append human-readable timestamps:

sudo /usr/sbin/tcplife-bpfcc -T

Troubleshooting Short-Lived Sockets and Keep-Alive Failures

Analysing tcplife output provides immediate clues for common architectural bottlenecks:

• Premature Socket Teardowns: Sockets closing in under 1ms with 0 KB transferred often indicate rapid connection resets (RST packets) caused by firewall rejections, SSL handshake mismatches, or full listen backlogs.
• Missing TCP Keep-Alives: Database connections remaining open for hours (e.g., > 3600000 ms) without active data exchange indicate idle leakages or unconfigured TCP keep-alive timers that consume file descriptors.
• Load Balancer Session Imbalances: Disparities in connection lifespans and byte ratios between web frontend instances help identify sticky session misconfigurations in reverse proxies.

By leveraging tcplife, Linux performance engineers and DevOps practitioners gain non-invasive, microsecond-accurate visibility into TCP connection lifecycles, enabling rapid isolation of network latency and transport inefficiencies.