How to Use the dropwatch Command in Linux to Diagnose Dropped Network Packets

In high-throughput Linux network engineering, diagnosing packet loss is notoriously difficult. When network latency spikes or connections intermittently time out, system administrators typically reach for familiar diagnostic utilities such as tcpdump, ping, or netstat -s. However, these tools suffer from a critical architectural blind spot: tcpdump only captures packets that successfully traverse the interface driver queue, while interface counters in ip -s link only increment generic aggregate drop numbers without indicating why or where inside the kernel a packet was discarded.

To eliminate this guesswork and locate the exact microscopic line of kernel code responsible for dropping a packet, Linux includes a specialised diagnostic tool: dropwatch. By interfacing directly with the kernel’s Network Drop Monitor (NET_DM) subsystem and listening to kfree_skb (Kernel Free Socket Buffer) tracepoints, dropwatch records every discarded packet and resolves the memory instruction pointer to the precise kernel function that called the free routine. This comprehensive guide covers the Linux network stack drop lifecycle, demonstrates how to configure and execute dropwatch, and explains how to decipher kernel symbol outputs to resolve packet loss.

Network Diagnostic Tooling: dropwatch vs. Traditional Utilities

Comparing dropwatch against standard Linux network analysis utilities illustrates why it is indispensable for root-cause diagnosis:

Diagnostic Tool	Kernel Stack Visibility	Shows Exact Drop Reason/Location	Performance Impact
dropwatch	Deep kernel-internal (captures kfree_skb tracepoints)	Yes (exact kernel function name and offset)	Minimal (lightweight netlink socket streaming)
tcpdump / wireshark	Network driver / AF_PACKET hook layer only	No (cannot see drops occurring inside IP/TCP logic)	Moderate to high (copies full packet payloads)
ip -s link / ethtool -S	Hardware NIC and driver ring buffers only	No (displays incrementing global counters only)	Negligible (reads hardware register counters)
bpftrace / eBPF	Universal kernel tracing	Yes (requires custom BPF scripts and BTF)	Minimal

Installing dropwatch and Loading Kernel Modules

The dropwatch utility is packaged in standard distribution repositories:

• Ubuntu / Debian: sudo apt install dropwatch
• Fedora / RHEL / AlmaLinux: sudo dnf install dropwatch
• Arch Linux: sudo pacman -S dropwatch

dropwatch requires the kernel’s network drop monitor driver. Ensure the kernel module is active:

sudo modprobe drop_monitor

Verify that the module loaded successfully:

lsmod | grep drop_monitor

Running dropwatch in Interactive Mode

Because dropwatch hooks directly into kernel tracepoints, elevated root privileges via sudo are mandatory. Launch the interactive session:

sudo dropwatch -l kas

The parameter -l kas instructs dropwatch to use /proc/kallsyms to automatically resolve raw hexadecimal instruction pointers into human-readable kernel function names.

Inside the interactive prompt, begin packet monitoring by typing start:

dropwatch> start
Enabling monitoring...
Kernel monitoring activated.
Issue Ctrl-C or 'stop' to halt monitoring.

As packets are dropped across your physical network interfaces, bridge interfaces, or loopback sockets, dropwatch streams drop events in real time:

14 drops at ip_rcv+0x60
2 drops at nf_hook_slow+0x7b
8 drops at tcp_v4_rcv+0xa2
1 drops at sk_stream_kill_queues+0x54

To pause collection and return to the prompt, type stop:

dropwatch> stop
Monitoring stopped.
dropwatch> exit

Deciphering Kernel Drop Locations

Each output line reports the number of discarded socket buffers alongside the specific kernel function where the discard occurred:

1. drops at nf_hook_slow / ipt_do_table / nft_do_chain

Root Cause: A firewall rule inside Netfilter (iptables or nftables) explicitly rejected or dropped the packet.

Action: Audit your active firewall chains using sudo iptables -L -v -n or sudo nft list ruleset to identify which rule counter is incrementing alongside the drops.

2. drops at ip_rcv

Root Cause: The incoming IP packet failed basic protocol integrity checks. Common triggers include:

• Malformed IP header checksums caused by NIC hardware offload bugs.
• Packets addressed to a destination IP not bound to any local interface on the host.
• MTU size mismatches where a packet exceeds the interface Maximum Transmission Unit and has the “Don’t Fragment” (DF) bit set.

3. drops at tcp_v4_rcv / tcp_v4_do_rcv

Root Cause: The packet reached the transport layer, but the TCP stack refused it. Common causes include:

• SYN packets arriving on a TCP port with no listening service (resulting in a TCP RST).
• Out-of-order packets or expired sequence numbers arriving outside the active TCP sliding window.
• TCP SYN flood protection dropping incoming requests when tcp_max_syn_backlog is exhausted.

4. drops at net_rx_action / enqueue_to_backlog

Root Cause: System CPU bottleneck. The kernel network receive queue (NAPI backlog) is completely full, meaning the operating system cannot process incoming packets from the hardware ring buffer fast enough.

Resolving Identified Bottlenecks via sysctl Tuning

Once dropwatch identifies the drop location, apply targeted kernel parameter adjustments in /etc/sysctl.conf:

If drops occur at enqueue_to_backlog, expand the kernel input queue backlog:

sudo sysctl -w net.core.netdev_max_backlog=10000

If drops occur during high-concurrency TCP connection handshakes:

sudo sysctl -w net.ipv4.tcp_max_syn_backlog=8192
sudo sysctl -w net.core.somaxconn=4096

Apply the changes permanently:

sudo sysctl -p

By leveraging dropwatch, network administrators replace speculative troubleshooting with precise, kernel-level evidence, allowing them to pinpoint firewall discards, queue overruns, and protocol errors in minutes.