How to Trace TCP Connection Queuing and Drops Using tcpdrop in Linux

Troubleshooting intermittent network connectivity issues, unexpected socket disconnects, or degraded application throughput requires visibility into kernel-level network packet handling. In Linux environments, packets dropped within the networking subsystem often go undetected by standard monitoring tools. The tcpdrop utility, part of the BPF Compiler Collection (BCC), provides real-time kernel instrumentation to identify when and why the Linux kernel drops TCP packets.

How tcpdrop Traces Kernel TCP Drops

The tcpdrop tool attaches eBPF probe points to the kernel function tcp_drop(). Whenever the Linux kernel intentionally discards an established or negotiating TCP packet—due to buffer exhaustion, invalid sequence numbers, checksum failures, state mismatch, or socket queue saturation—tcpdrop intercepts the event and prints detailed socket telemetry including source and destination IP addresses, ports, TCP state, and the exact kernel call stack trace.

Installing the BCC Tracing Utilities

Before running tcpdrop, ensure that the BCC tools and Linux kernel development headers are installed on your system:

# On Debian or Ubuntu systems:
sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

# On RHEL, Rocky Linux, or Fedora systems:
sudo dnf install bcc-tools kernel-devel-$(uname -r)

Once installed, the utility is available as tcpdrop-bpfcc on Debian/Ubuntu or /usr/share/bcc/tools/tcpdrop on RHEL-based distributions.

Executing and Interpreting tcpdrop Output

Run tcpdrop with elevated administrative privileges to start capturing kernel drop events in real time:

sudo tcpdrop-bpfcc -s

The -s flag instructs the tool to print kernel stack traces for each dropped packet, which is critical for identifying the exact kernel module or network stack logic causing the rejection.

Example terminal output:

TIME     PID     IP SADDR:SPORT          DADDR:DPORT          STATE       FLAGS
10:14:02 0       4  192.168.1.100:443    192.168.1.50:54320   ESTABLISHED ACK
        tcp_drop+0x1
        tcp_rcv_established+0x1d5
        tcp_v4_do_rcv+0x142
        tcp_v4_rcv+0xca0
        ip_protocol_deliver_rcu+0x3c
        ip_local_deliver_finish+0x44

Diagnosing Root Causes from Kernel Call Stacks

By analysing the kernel stack trace reported by tcpdrop, system engineers can immediately differentiate between common networking bottlenecks:

• Listen Queue Overflow: If the stack trace points to tcp_v4_syn_recv_sock or reqsk_queue_alloc, the application listen backlog is full. Mitigate this by increasing net.core.somaxconn and tuning your web or application server socket backlogs.
• Invalid State or Out-of-Window Packets: Drops occurring within tcp_validate_incoming or tcp_rcv_state_process indicate packets received outside the valid sliding TCP window, often caused by asymmetrical routing or aggressive middlebox connection pruning.
• Memory Pressure: Stacks ending in __alloc_skb or socket buffer exhaustion indicate kernel memory thresholds reached under net.ipv4.tcp_rmem or net.ipv4.tcp_wmem.

Using tcpdrop provides definitive, low-overhead diagnostic evidence to isolate and resolve complex network performance anomalies across Linux enterprise workloads.