How to Trace Inbound TCP Connections and Backlog Delays in Linux Using tcpaccept

When high-throughput Linux servers—such as NGINX reverse proxies, Redis caches, Envoy service meshes, or PostgreSQL clusters—experience latency spikes or connection refusals, standard diagnostic tools often fail to capture transient anomalies. Commands like netstat or ss report queue sizes only at the instant of execution, missing brief traffic micro-bursts that saturate listener sockets. Furthermore, packet capture tools like tcpdump generate massive overhead on saturated interfaces. In production Linux environments, observing inbound connection arrivals and listener responsiveness requires kernel-level instrumentation. Using the eBPF-based tcpaccept tool from the BPF Compiler Collection (BCC), system administrators and performance engineers can trace every passive TCP connection established via the accept() syscall, monitor listener backlog queue delays, and attribute traffic flows to application processes in real time.

Understanding Inbound TCP Connection Handshakes and Accept Queues

In the Linux networking stack, passive TCP connection establishment involves two distinct kernel queues before the application process retrieves the connection:

• The SYN Queue (Incomplete Connection Queue): When a client sends a TCP SYN packet, the kernel replies with a SYN-ACK and allocates a request socket in the SYN queue. If this queue overflows (due to SYN floods or high connection velocity), incoming SYNs are dropped or SYN cookies are engaged.
• The Accept Queue (Complete Connection Queue): Once the client returns the final ACK, the three-way handshake completes. The socket transitions to the ESTABLISHED state and enters the accept queue, waiting for the application process to invoke the accept() or accept4() system call.
• Backlog Queue Delay: The duration a fully established socket waits in the accept queue before being accepted by the application thread represents backlog latency. High backlog latency indicates application thread starvation, worker process saturation, or blocking I/O within the service.
• eBPF Tracepoint Instrumentation: Rather than intercepting network packets on the wire, tcpaccept hooks into the kernel’s inet_csk_accept() function using kprobes or fentry tracepoints. It extracts metadata directly from kernel socket data structures as connections are passed to user space.

Installing BCC Tools and Kernel Headers

Running tcpaccept requires administrative root privileges and kernel development headers to compile BPF bytecode or validate CO-RE BTF information.

On modern Ubuntu and Debian platforms, install the standard BCC package suite:

sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

On RHEL, Rocky Linux, or Fedora installations, enable BCC packages via dnf:

sudo dnf install bcc-tools kernel-devel-$(uname -r)

Verify that the tool is accessible in your execution path:

which tcpaccept || which /usr/sbin/tcpaccept-bpfcc

On Debian and Ubuntu, BCC utilities append the -bpfcc suffix by default.

Real-Time Inbound TCP Connection Tracing

To begin monitoring incoming TCP connections across all running server applications, execute tcpaccept with administrative privileges:

sudo tcpaccept-bpfcc

As remote clients connect and services call accept(), live tabular telemetry streams to the terminal:

PID     COMM         IP RADDR            RPORT LADDR            LPORT
18420   nginx        4  198.51.100.45    52114 10.0.1.15        443
18420   nginx        4  203.0.113.88     49812 10.0.1.15        443
2104    postgres     4  10.0.2.50        38910 10.0.1.15        5432
1405    sshd         6  [2001:db8::2]    60124 [2001:db8::1]    22

Each field reveals vital connection metrics:

• PID: The process identifier that invoked accept().
• COMM: The executable command name (e.g. nginx, postgres, sshd).
• IP: IP protocol version (4 for IPv4, 6 for IPv6).
• RADDR:RPORT: The remote client IP address and ephemeral source port.
• LADDR:LPORT: The local interface IP address and server listening port.

Measuring Listener Backlog Latency and Timestamps

In high-performance API gateways and database clusters, measuring how long connections sit in the kernel accept queue before the application processes them helps identify application thread bottlenecks.

To display high-resolution timestamps alongside each accepted connection (useful when correlating connection spikes with application error logs):

sudo tcpaccept-bpfcc -t

To include container network namespace information (cgroup IDs) when auditing containerized services under Docker or Kubernetes:

sudo tcpaccept-bpfcc -c

For custom latency histograms measuring backlog queue duration, engineers can execute a targeted bpftrace script to track the elapsed time between socket establishment and the return of inet_csk_accept():

sudo bpftrace -e 'kretprobe:inet_csk_accept /retval/ { @latency_us = hist(nsecs / 1000); }'

Filtering by Listening Port and Process ID

On production servers handling thousands of requests per second, tracing all connections can flood the terminal. Command-line flags allow targeted filtering.

To trace only incoming connections directed to specific listening ports (such as HTTP port 80 and HTTPS port 443):

sudo tcpaccept-bpfcc -p 80,443

To isolate connections accepted by a specific server process (for example, auditing an isolated database instance with PID 2104):

sudo tcpaccept-bpfcc -P 2104

To trace exclusively IPv6 connections while ignoring legacy IPv4 traffic, pass the -6 flag:

sudo tcpaccept-bpfcc -6

Tuning Linux Kernel Accept Backlog Limits

When tcpaccept reveals high connection velocity or application logs report connection timeouts, verify that the kernel and application backlog limits are tuned appropriately.

Inspect the system-wide maximum backlog limit in sysctl:

sysctl net.core.somaxconn
sysctl net.ipv4.tcp_max_syn_backlog

If high-concurrency web servers drop incoming connections during traffic surges, increase the socket listen backlog and SYN queue limits:

sudo sysctl -w net.core.somaxconn=65535
sudo sysctl -w net.ipv4.tcp_max_syn_backlog=65535

Ensure application listener directives reflect the elevated kernel limits. For example, in NGINX configuration files, declare the backlog parameter explicitly on the listen directive:

listen 443 ssl http2 default_server backlog=65535;

By monitoring inbound connections with tcpaccept, administrators verify listener health, isolate worker thread starvation, and ensure reliable service delivery across Linux production clusters.