How to Trace Outbound TCP Connection Attempts and Latency in Linux Using tcpconnect

Modern microservice architectures, cloud-native deployments, and distributed systems depend on continuous outbound network communication—querying remote databases, communicating with third-party payment gateways, resolving external REST APIs, and synchronizing cache clusters. When applications experience intermittent timeouts or unexpected connection refusal errors, systems administrators need to know exactly which processes are initiating outbound TCP connections, where they are attempting to connect, and how long the three-way handshake requires to complete. The BCC eBPF tool tcpconnect traces every outbound TCP connection in real time.

Understanding How tcpconnect Instruments the Linux TCP Stack

Standard network utilities like netstat or ss report existing active sockets or listening endpoints. However, short-lived outbound connections (such as microservice health checks or transient API calls lasting only a few milliseconds) terminate before polling tools can capture them.

tcpconnect instruments the Linux kernel’s internal TCP connection initiation path via eBPF:

• Kernel Probe Hook: Attaches to the kernel function tcp_v4_connect() (and tcp_v6_connect() for IPv6), which is called whenever a local socket initiates an outbound connect() system call.
• Zero Polling Overhead: Emits telemetry only when an active connection is initiated, eliminating the CPU waste associated with polling /proc/net/tcp.
• Latency Measurement: When invoked with the -d or -L flag, tcpconnect tracks the time elapsed between the initial outbound SYN packet and the arrival of the server’s SYN-ACK, measuring the precise TCP three-way handshake round-trip latency.

Installing the BCC Tools Suite

The tcpconnect tool is part of the BPF Compiler Collection (BCC):

# Ubuntu / Debian
sudo apt-get update
sudo apt-get install -y bpfcc-tools linux-headers-$(uname -r)

# RHEL 9 / Rocky Linux 9 / AlmaLinux
sudo dnf install -y bcc-tools kernel-devel-$(uname -r)

# Verify executable path
which tcpconnect || which /usr/share/bcc/tools/tcpconnect

Running tcpconnect to Capture Outbound Sockets

To begin real-time tracing of all outbound TCP connections across the operating system, launch tcpconnect with root privileges:

# Basic real-time outbound connection tracing
sudo /usr/share/bcc/tools/tcpconnect

# Trace outbound connections and measure handshake latency (in ms)
sudo /usr/share/bcc/tools/tcpconnect -d

# Filter connections initiated by a specific process ID (PID 14220)
sudo /usr/share/bcc/tools/tcpconnect -p 14220

# Filter connections targeting a specific remote destination port (e.g. port 443)
sudo /usr/share/bcc/tools/tcpconnect -P 443

Interpreting Output Columns and Latency Metrics

When an application attempts an outbound connection, tcpconnect outputs a real-time event record:

TIME(s)  PID     COMM         IP SADDR            DADDR            DPORT LAT(ms)
05:42:08 28140   curl         4  192.168.1.100    142.250.180.14   443   14.22
05:42:15 1192    redis-server 4  127.0.0.1        127.0.0.1        6379  0.12
05:42:22 30104   python3      4  192.168.1.100    10.0.4.55        5432  1.85
05:42:38 31088   node         4  192.168.1.100    93.184.216.34    80    48.91

The telemetry reveals essential networking indicators:

• PID and COMM: The process identifier and executable responsible for initiating the connection.
• IP: IP version (4 or 6).
• SADDR: Local source IP address selected by kernel routing.
• DADDR: Destination IP address being contacted.
• DPORT: Destination port number (e.g. 443 for HTTPS, 5432 for PostgreSQL, 6379 for Redis).
• LAT(ms): Elapsed time required to complete the three-way SYN/SYN-ACK handshake.

Identifying Rogue Network Calls and Handshake Latency Spikes

Security engineers and infrastructure architects can leverage tcpconnect for forensic auditing:

1. Detecting Unintended External Dependencies: If an internal containerized service unexpectedly initiates connections to external IP addresses on port 80 or 443, the application may be leaking telemetry or executing malicious reverse shells.
2. Diagnosing WAN Degradation: If local database handshakes (10.0.4.55:5432) take 1-2 ms while external API calls take 200+ ms, the latency is situated in upstream ISP routing rather than local server contention.
3. Detecting Port Scans and Retries: Bursts of rapid consecutive connection attempts from a single process indicate retry storms or connection pool starvation.

Using tcpconnect provides systems engineers with immediate visibility into outbound network activity, establishing rapid root-cause analysis for distributed service latencies and network connectivity failures.