How to Trace TCP Queue Lengths and Backlog Saturation Using tcpqlen in Linux

High-concurrency network servers—such as reverse proxies, API gateways, and web application servers—can suffer from sudden latency degradation when network queues become congested. In the Linux TCP subsystem, two distinct queuing mechanisms dictate flow control: the established connection send/receive queues (managed within socket buffers) and the listen backlog queue (holding pending SYN/ACK connections). Monitoring queue depth over time reveals buffer bloat and queue saturation that aggregate traffic statistics conceal. The tcpqlen tool from the BCC toolkit samples TCP queue lengths with minimal overhead.

Understanding Socket Queues and Backlog Saturation

Queue congestion impacts application performance in specific ways:

  • Transmit Queue (T-Q) Saturation: Indicates that the local application is generating data faster than the network interface can transmit or the remote client can acknowledge, consuming kernel socket buffer memory (wmem).
  • Receive Queue (R-Q) Saturation: Signifies that incoming packets are arriving over the network faster than the user-space application thread can read them with recv(), leading to window scaling stalls.
  • Listen Backlog Overflows: Occurs when incoming SYN packets exceed the application’s listen queue limit (listen(fd, backlog)), resulting in dropped connection handshakes.

Installing the BCC Tracing Tools

Ensure that BCC tools and kernel development headers are installed on your Linux distribution:

# On Debian or Ubuntu systems:
sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

# On RHEL, Rocky Linux, or Fedora:
sudo dnf install bcc-tools kernel-devel-$(uname -r)

On Debian/Ubuntu environments, run the utility as tcpqlen-bpfcc; on enterprise RHEL distributions, it is located at /usr/share/bcc/tools/tcpqlen.

Running tcpqlen to Sample TCP Queue Depths

To inspect TCP queue depths across the system, execute tcpqlen with administrative privileges:

sudo tcpqlen-bpfcc

Example terminal output:

TIME(s)     COMM           PID    LADDR:LPORT          RADDR:RPORT          T-Q    R-Q
0.000000    nginx          2104   192.168.1.50:443     203.0.113.12:48910   0      14280
0.051200    postgres       1842   192.168.1.50:5432    10.0.0.10:38120     65536  0
0.102400    node           3100   127.0.0.1:3000       127.0.0.1:41200      128    0

Analysing Queue Metrics for System Tuning

Evaluating the output identifies distinct bottlenecks:

  • High R-Q (Receive Queue): High byte counts in R-Q on web servers (e.g., nginx) indicate that the application worker process is blocked on CPU computation, database queries, or disk I/O, leaving incoming client payloads stalled in kernel memory.
  • High T-Q (Transmit Queue): Large numbers in T-Q highlight outbound bandwidth throttling, slow remote client download speeds, or asymmetric network routes.

Remediation Strategies for Saturated Queues

Based on tcpqlen diagnostic data, apply appropriate kernel and application optimizations:

  • Increase System Socket Backlog: Raise the maximum allowable socket listen backlog in sysctl:
    sudo sysctl -w net.core.somaxconn=4096
    sudo sysctl -w net.ipv4.tcp_max_syn_backlog=4096
  • Enable TCP Fast Open and Syncookies: Protect against backlog queue exhaustion caused by connection floods:
    sudo sysctl -w net.ipv4.tcp_syncookies=1
  • Application Worker Tuning: Increase the number of asynchronous event loops or multi-threaded worker processes to consume incoming receive buffers before queues saturate.

Sampling queue lengths with tcpqlen provides definitive kernel-level insight, enabling administrators to eliminate buffer bloat and maintain steady throughput on Linux servers.

Get the best tech tips delivered straight to your inbox.

Join thousands of readers mastering Apple, Google, Microsoft, and Linux.