How to Trace Active TCP Connections and Connection Latency in Linux Using tcpconnect

When troubleshooting microservice communication bottlenecks, distributed application timeouts, or unauthorized outbound data exfiltration, system administrators require immediate visibility into new outbound TCP sockets. Traditional tools like netstat, ss, or lsof sample already established or lingering sockets, periodically missing ephemeral connections that complete in milliseconds. Conversely, full packet inspection with tcpdump introduces substantial CPU overhead and generates massive capture files on saturated multi-gigabit interfaces. In modern production environments, tracing new outbound TCP connections efficiently requires eBPF instrumentation. Using the tcpconnect tool from the BPF Compiler Collection (BCC), engineers can capture every active connect() syscall in real time, record connection latency, and attribute network activity directly to source processes.

Understanding Active TCP Connection Tracing with eBPF

In the Linux networking subsystem, an active TCP connection begins when an application process executes the connect() system call, prompting the kernel to dispatch a TCP SYN segment. Key architectural mechanisms traced by tcpconnect include:

• Kernel Probe Attachment: tcpconnect hooks into the kernel’s tcp_v4_connect() and tcp_v6_connect() functions using kprobes or fentry tracepoints. It extracts the calling process ID (PID), command name (COMM), and socket address tuple before the SYN packet traverses the network interface.
• Handshake Latency Measurement: When the -d or -L flag is enabled, tcpconnect tracks the time elapsed between the outbound SYN and the reception of the matching SYN-ACK segment, capturing round-trip handshake latency across WAN or LAN boundaries.
• Zero Polling Overhead: Unlike shell scripts executing continuous ss loops, eBPF probes remain dormant until an active connection is initiated. The kernel executes the compiled in-kernel BPF bytecode only upon socket state transitions.
• Container and Namespace Visibility: Because probes attach at the host kernel level, tcpconnect monitors connection requests initiated inside Docker containers, Kubernetes pods, and network namespaces without requiring agent sidecars.

Prerequisites and BCC Tools Installation

Running tcpconnect requires root privileges and a Linux kernel compiled with eBPF support (Linux 4.9 or later, with standard BTF enabled on modern kernels).

On modern Ubuntu and Debian platforms, install the BCC package suite:

sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

On RHEL, AlmaLinux, or Fedora hosts, install the required packages via dnf:

sudo dnf install bcc-tools kernel-devel-$(uname -r)

Confirm the executable is accessible in your system path:

which tcpconnect || which /usr/sbin/tcpconnect-bpfcc

On Debian and Ubuntu, BCC utilities append the -bpfcc suffix to avoid namespace collisions with legacy binaries.

Real-Time Outbound TCP Connection Monitoring

To start tracing active outbound TCP connections across the host in real time, execute tcpconnect with administrative elevation:

sudo tcpconnect-bpfcc

As applications, background daemons, or container workloads initiate connections, the utility streams live tabular data to standard output:

PID    COMM         IP SADDR            DADDR            DPORT
28410  curl         4  192.168.1.50     93.184.216.34    80
28412  ssh          4  192.168.1.50     10.0.0.15        22
1402   dockerd      6  [2001:db8::1]    [2001:db8::2]    443
28450  redis-cli    4  127.0.0.1        127.0.0.1        6379

The output fields provide immediate diagnostic clarity:

• PID: The process identifier that invoked connect().
• COMM: The executable command name (e.g. curl, dockerd, python3).
• IP: IP protocol version (4 for IPv4, 6 for IPv6).
• SADDR: The source IP address bound to the local interface or loopback.
• DADDR: The destination host IP address.
• DPORT: The remote destination port.

Measuring TCP Handshake Latency and Timestamps

In distributed database topologies and microservice clusters, understanding connection establishment latency helps detect network congestion or overloaded remote listeners.

To measure the duration of the TCP three-way handshake (time to establish connection in milliseconds), pass the -d flag:

sudo tcpconnect-bpfcc -d

The generated output adds a dedicated LAT(ms) column:

PID    COMM         IP SADDR        DADDR            DPORT LAT(ms)
29104  node         4  10.0.1.25    10.0.2.100       5432  0.42
29150  app-worker   4  10.0.1.25    198.51.100.12    443   48.15

To include high-resolution timestamps with each event for correlation with centralized logging platforms, include the -t flag:

sudo tcpconnect-bpfcc -t

Filtering Connections by Port and Process ID

On high-concurrency systems, unfiltered tracing can generate overwhelming amounts of data. Use command-line arguments to isolate specific applications or destination services.

To monitor only connections directed to specific destination ports (such as HTTPS 443 and DNS 53):

sudo tcpconnect-bpfcc -p 443,53

To restrict tracing to a specific process ID (for example, auditing an isolated worker process with PID 1845):

sudo tcpconnect-bpfcc -P 1845

To trace only IPv6 connections while filtering out legacy IPv4 traffic, use the -6 flag:

sudo tcpconnect-bpfcc -6

To include container network namespace information for Kubernetes pods, pass the -U flag to print UID attributes:

sudo tcpconnect-bpfcc -U

Correlating Handshake Delays with Network Health

Analyzing handshake latency patterns enables engineers to quickly isolate root causes:

1. Uniform High Latency (LAN Workloads): If local microservice connections report latency exceeding 10 ms across internal switches, check for virtual switch contention, CPU starvation in hypervisors, or duplex mismatches.
2. Intermittent Latency Spikes (WAN Endpoints): Occasional multi-second handshake delays typically signify SYN packet drops requiring retransmissions, often caused by upstream stateful firewall table exhaustion or aggressive SYN-flood mitigation thresholds.
3. Rapid Connect Refusals: If tcpconnect logs rapid connection terminations with near-zero latency followed immediately by application retry loops, the remote service port is closed (RST packet returned) or intermediate security policies reject the handshake.

By using tcpconnect, infrastructure engineers gain instantaneous, low-overhead observability into outbound TCP connections, accelerating root-cause analysis for modern Linux systems.