How to Trace TCP Connection Latency and SYN-ACK Delays in Linux Using tcpconnlat

When microservices, web applications, or database clients experience slow response times, determining whether the bottleneck stems from network transport delays or application processing latencies is critical. In distributed systems, high latency during the initial TCP three-way handshake (the interval between sending a SYN packet and receiving a SYN-ACK response) indicates WAN packet loss, routing congestion, or saturated firewall state tables. By contrast, delays occurring after connection establishment point directly to application-layer query execution overhead. To measure outbound handshake round-trip times with microsecond precision and zero packet-capture overhead, the BPF Compiler Collection (BCC) provides tcpconnlat.

How tcpconnlat Leverages eBPF for Handshake Tracing

Unlike packet capture sniffers (such as tcpdump) that copy complete packet payloads across the kernel-userspace boundary, tcpconnlat attaches lightweight Extended Berkeley Packet Filters (eBPF) directly to kernel socket functions:

• tcp_v4_connect() / tcp_v6_connect(): When an outbound connection is initiated via the connect() system call, tcpconnlat stores a high-resolution timestamp in an internal BPF map indexed by the kernel struct sock pointer.
• tcp_finish_connect(): When the remote peer responds with a SYN-ACK and the socket completes the handshake, a secondary kprobe retrieves the initial timestamp, calculates the delta in milliseconds, and emits a structured event line to userspace.
• Map Eviction: The socket entry is immediately purged from the BPF hash table, guaranteeing minimal memory footprint even on servers processing thousands of connections per second.

Installing tcpconnlat in Linux

The tcpconnlat utility is included within the BCC tracing package. On Ubuntu and Debian systems, install the toolset using apt:

sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

On RHEL, Rocky Linux, AlmaLinux, or Fedora:

sudo dnf install bcc-tools kernel-devel-$(uname -r)

The executable is typically installed as /usr/sbin/tcpconnlat-bpfcc or located in /usr/share/bcc/tools/tcpconnlat.

Tracing Active Outbound Connection Latencies

Execute tcpconnlat with root privileges to trace all outbound TCP handshakes in real time:

sudo /usr/sbin/tcpconnlat-bpfcc

When processes initiate outbound network requests, tcpconnlat outputs formatted diagnostic records:

PID    COMM         IP SADDR            DADDR            DPORT LAT(ms)
14820  curl         4  192.168.1.10     93.184.216.34    443   18.42
15104  node         4  192.168.1.10     10.0.0.50        5432  0.45
15220  python3      6  2001:db8::10     2001:db8::80     443   124.80
15340  redis-cli    4  192.168.1.10     10.0.0.60        6379  0.22

Key output columns provide essential diagnostic indicators:

• PID / COMM: The process identifier and executable name initiating the socket connection.
• IP: IP protocol version (4 for IPv4, 6 for IPv6).
• SADDR / DADDR: Local source IP address and destination server IP.
• DPORT: Destination service port (e.g., 443 for HTTPS, 5432 for PostgreSQL).
• LAT(ms): The exact round-trip duration of the three-way handshake in milliseconds.

Filtering by Latency Threshold and Process

In high-traffic production environments, logging every sub-millisecond local connection creates unnecessary terminal noise. You can filter tcpconnlat to focus strictly on problematic connections:

1. Filter by Minimum Latency Threshold (e.g., Handshakes Slower than 50ms)

To identify slow external API endpoints or laggy database nodes, append a minimum latency threshold argument:

sudo /usr/sbin/tcpconnlat-bpfcc 50

Only handshakes taking 50ms or longer will be displayed, immediately isolating troubled network paths.

2. Filter by Specific Process ID (PID)

To trace handshake performance exclusively for a specific application daemon (e.g., PID 4120):

sudo /usr/sbin/tcpconnlat-bpfcc -p 4120

3. Include Absolute Timestamps

To correlate latency spikes with application error logs and monitoring metrics, include high-resolution timestamps:

sudo /usr/sbin/tcpconnlat-bpfcc -T

Diagnosing Network vs Application Bottlenecks

Analysing tcpconnlat metrics alongside application performance logs provides definitive architectural insights:

• High Connection Latency (LAT > 100ms): Indicates physical distance, inter-region routing delays, ISP peering issues, or SYN packet drops triggering TCP exponential backoff retransmissions before the first SYN-ACK arrives.
• Low Handshake Latency (< 2ms) with Slow API Responses: If tcpconnlat confirms a sub-millisecond connection handshake to your database or microservice backend, but queries take 2,000ms to return data, the delay is 100% application-bound (e.g., lock contention, slow disk I/O, or unindexed queries).

Leveraging tcpconnlat equips Linux performance engineers with microsecond-accurate network transport telemetry, eliminating guesswork and accelerating root-cause analysis across distributed cloud infrastructures.