How to Trace Linux Kernel TCP Packet Drops and Drop Stacks Using tcpdrop

In high-throughput Linux networking environments, intermittent packet loss represents one of the most frustrating performance defects to isolate. When an application reports sudden socket timeouts or degraded throughput, traditional network diagnostics like ping or ethtool report only interface-level physical statistics. They confirm that drops occurred, but reveal nothing about why or where inside the operating system kernel the packets were discarded. The Linux kernel drops TCP packets for numerous internal reasons: socket buffer exhaustion, invalid TCP checksums, sequence number mismatches, stateful firewall (iptables/nftables) rejections, or TCP window collapses. Using the eBPF-based tcpdrop utility from the BPF Compiler Collection (BCC), system administrators and performance engineers can trace TCP packet drops in real time, capturing socket tuples, TCP header states, and the exact kernel call stack responsible for the drop.

Understanding Linux Kernel TCP Packet Drop Mechanics

When a network packet arrives on an Ethernet interface, it traverses the driver ring buffer, the network interface layer, netfilter hooks, and the TCP protocol handler before reaching user space. tcpdrop instruments key drop points across this pipeline:

• Kernel Function Hooking: tcpdrop attaches eBPF kprobes directly to the kernel’s internal tcp_drop() and kfree_skb() functions. Whenever the networking stack drops an active or initializing TCP segment, the probe intercepts the event instantly.
• Kernel Call Stack Extraction: Beyond reporting IP addresses and ports, tcpdrop records the kernel instruction pointer and stack trace (kernel backtrace). This exposes the exact kernel C function (e.g. tcp_v4_rcv, tcp_prune_queue, nf_hook_slow) executing the discard.
• TCP State Context: The tool captures connection state flags (SYN_SENT, ESTABLISHED, TIME_WAIT) alongside TCP sequence numbers, helping distinguish between network attacks, firewall drops, and buffer starvation.
• Zero In-Band Packet Overhead: Unlike tcpdump, which copies every packet across the user-kernel boundary, tcpdrop executes only when a packet drop occurs, ensuring negligible system impact even on multi-gigabit links.

Installing BCC Tools and Kernel Headers

Running tcpdrop requires root administrative privileges and kernel development headers to compile eBPF programs or resolve kernel symbols.

On modern Ubuntu and Debian platforms, install the standard BCC package suite:

sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

On RHEL, AlmaLinux, or Fedora installations, enable BCC packages via dnf:

sudo dnf install bcc-tools kernel-devel-$(uname -r)

Confirm the utility is accessible in your system execution path:

which tcpdrop || which /usr/sbin/tcpdrop-bpfcc

On Debian and Ubuntu, BCC utilities append the -bpfcc suffix by default.

Real-Time TCP Drop Monitoring and Call Stack Capture

To begin monitoring TCP packet drops across the entire Linux operating system, execute tcpdrop with administrative elevation:

sudo tcpdrop-bpfcc

When the kernel drops a TCP segment, the tool prints the event details followed immediately by the kernel call stack:

TIME     PID    IP SADDR:SPORT          DADDR:DPORT          STATE       FLAGS
05:55:12 0      4  198.51.100.45:52114  10.0.1.15:443        ESTABLISHED ACK
        tcp_v4_do_rcv+0x94
        tcp_v4_rcv+0xb25
        ip_protocol_deliver_rcu+0x3c
        ip_local_deliver_finish+0x44
        ip_local_deliver+0xe0
        ip_sublist_rcv_finish+0x37
        ip_sublist_rcv+0x180
        __netif_receive_skb_core+0x2c0
        netif_receive_skb_list_internal+0x1a0
        napi_complete_done+0x72

The output fields provide deep technical insight:

• TIME: Wall-clock timestamp of the drop event.
• PID: Process ID associated with the socket (or 0 if dropped in hardware interrupt context before reaching an application thread).
• SADDR:SPORT / DADDR:DPORT: The remote and local IP address and port tuples.
• STATE: The internal TCP connection state when the drop occurred.
• Kernel Stack Trace: The hierarchy of internal kernel functions leading to the drop.

Analyzing Root Causes from Kernel Call Stacks

Interpreting the top function in the kernel call stack enables engineers to diagnose the root cause immediately:

1. nf_hook_slow / ipt_do_table: The packet was intentionally dropped by a netfilter firewall rule (such as iptables, nftables, or ufw). Inspect active firewall policies or connection tracking limits (nf_conntrack_max).
2. tcp_prune_queue / tcp_try_rmem_schedule: The socket’s receive buffer is exhausted. The application process is reading data too slowly from the socket, causing the kernel to drop incoming packets due to buffer exhaustion.
3. tcp_v4_rcv (Out-of-Window / Sequence Errors): The packet arrived with sequence or acknowledgement numbers that fall outside the current sliding window, often caused by asymmetrical routing or delayed duplicate segments on WAN links.
4. tcp_v4_conn_request (Listen Backlog Drop): The server’s accept backlog or SYN queue is full, causing new incoming connection handshakes to be discarded.

Capturing Drops with Timestamps and Custom bpftrace Probes

To record packet drops with millisecond timestamps for correlation with centralized logging systems, invoke the -t flag:

sudo tcpdrop-bpfcc -t

For bespoke filtering (e.g. logging only packet drops directed to HTTPS port 443 while printing the drop reason code), use a lightweight bpftrace one-liner:

sudo bpftrace -e 'kprobe:tcp_drop { $sk = (struct sock *)arg0; printf("Drop on port %d, state %d\n", $sk->__sk_common.skc_num, $sk->__sk_common.skc_state); }'

Tuning Kernel Buffers to Mitigate Socket Drops

When tcpdrop indicates that drops are caused by socket memory pressure (e.g. tcp_try_rmem_schedule), tune the TCP memory allocation limits via sysctl.

Inspect the current TCP receive and transmit buffer boundaries:

sysctl net.ipv4.tcp_rmem
sysctl net.core.rmem_max

To eliminate socket buffer drops on high-throughput 10GbE interfaces, expand maximum receive memory limits:

sudo sysctl -w net.core.rmem_max=16777216
sudo sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"

Running tcpdrop before and after applying buffer changes provides objective kernel verification that packet drops have been completely eliminated.