Troubleshooting intermittent network latency on high-throughput Linux servers often requires visibility into how quickly the kernel hands off newly established TCP connections to listening user-space daemons. While standard socket utilities such as ss or netstat display current socket counts and queue depths, they cannot measure the precise time elapsed between connection establishment and the accept() system call. The tcpaccept utility, part of the BCC (BPF Compiler Collection) and bpftrace toolsets, leverages extended Berkeley Packet Filters (eBPF) to trace passive TCP connections in real time without imposing noticeable performance overhead.
Understanding Passive TCP Connections and Accept Queues
When an external client initiates a TCP handshake, the Linux kernel processes the incoming SYN packet, responds with a SYN-ACK, and places the pending connection into the SYN backlog (incomplete connection queue). Upon receiving the final ACK, the kernel transitions the socket to the established state and moves it into the accept queue (listen backlog).
The connection remains in this queue until the server process (such as Nginx, Apache, or a custom microservice) invokes the accept() or accept4() system call. If the application thread is blocked, experiencing high garbage collection pauses, or constrained by CPU scheduling latency, incoming connections accumulate in the accept queue. The tcpaccept tool hooks into the kernel’s tcp_accept() routines to record each completed connection along with its duration in the queue.
Prerequisites and Tool Installation
To run tcpaccept, your Linux host must have an eBPF-capable kernel (Linux 4.9 or higher, with Linux 5.4+ recommended) and root privileges. Install the BPF Compiler Collection package corresponding to your distribution:
Ubuntu and Debian
sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)
On Debian systems, BCC binaries are installed into /usr/sbin/ with a -bpfcc suffix (e.g., /usr/sbin/tcpaccept-bpfcc).
RHEL, Rocky Linux, and Fedora
sudo dnf install bcc-tools kernel-devel-$(uname -r)
On RHEL-based distributions, tools are located under /usr/share/bcc/tools/.
Tracing Inbound Connections in Real Time
To capture passive connections across all listening ports on your server, execute tcpaccept with superuser privileges:
sudo tcpaccept-bpfcc
The tool outputs tabular columns capturing active connection details:
TIME PID COMM IP RADDR RPORT LADDR LPORT
05:14:22 1842 nginx 4 203.0.113.45 54210 192.0.2.10 443
05:14:22 1842 nginx 4 198.51.100.8 49152 192.0.2.10 443
05:14:23 2910 node 4 127.0.0.1 38194 127.0.0.1 3000
The column headers indicate:
- TIME: Time of day when the
accept()call completed. - PID: Process identifier of the listening application.
- COMM: Command name of the receiving process.
- IP: IP protocol version (4 or 6).
- RADDR / RPORT: Remote client IP address and ephemeral source port.
- LADDR / LPORT: Local listening IP address and destination port.
Measuring Accept Queue Latency
By default, tcpaccept lists connections without calculating queue dwell time. To measure how long sockets spent in the kernel accept queue before the user-space process retrieved them, supply the -l (latency) flag:
sudo tcpaccept-bpfcc -l
This adds a LAT(ms) column to the output stream:
TIME PID COMM IP RADDR RPORT LADDR LPORT LAT(ms)
05:15:01 1842 nginx 4 203.0.113.88 51204 192.0.2.10 443 0.12
05:15:02 3411 python3 4 10.0.0.15 48922 10.0.0.2 8080 45.89
In this example, Nginx handles handshakes in 0.12 ms, whereas the Python application exhibits an accept queue latency of nearly 46 ms, indicating thread contention or event-loop starvation.
Filtering by Specific Ports and Timestamps
In production environments handling thousands of concurrent connections, unfiltered output can saturate your terminal. You can filter monitoring to target specific service ports:
sudo tcpaccept-bpfcc -p 443 -l
To include high-resolution millisecond timestamps for correlating connection bursts with application log entries, append the -T flag:
sudo tcpaccept-bpfcc -T -l
Diagnostic Interpretations and Remediation
- Elevated LAT(ms) with Normal CPU Load: Indicates application thread starvation or synchronous blocking operations inside the worker loop. Review worker thread pool sizing and asynchronous I/O settings.
- Dropped Connections (SYN Flood / Listen Overflow): Correlate
tcpacceptlatency spikes with kernel queue drop counters usingnetstat -s | grep -i listen. - Tuning Listen Backlogs: If the application cannot accept connections quickly enough during traffic spikes, inspect the system-wide queue limit via
sysctl net.core.somaxconnand increase the backlog parameter in your application configuration.