How to Use the Linux bpftrace Command to Monitor Network Packet Drops

The Challenge of Network Packet Drops

In high-throughput Linux environments, network performance issues are notoriously difficult to diagnose. When a web server experiences high latency or connections simply timeout, the underlying cause is often packet dropping—the Linux kernel intentionally discarding incoming or outgoing network packets. This can occur due to full socket buffers, aggressive firewall rules (iptables/nftables), or exhausted network interface rings.

Traditional tools like netstat or ifconfig can tell you that packets were dropped, but they cannot tell you why. To discover exactly which function in the Linux kernel decided to discard a packet, administrators must use bpftrace, a dynamic tracing tool built on top of the eBPF framework.

Prerequisites

To use bpftrace, you need a modern Linux distribution (Ubuntu 22.04/24.04, RHEL 8/9) with the appropriate eBPF toolchain and kernel headers installed.

sudo apt update
sudo apt install bpftrace linux-headers-$(uname -r) -y

Because bpftrace interacts directly with internal kernel structures, you must run it with root privileges.

Step 1: Understanding kprobes and tracepoints

bpftrace works by attaching small monitoring scripts to specific events in the kernel, known as kprobes (kernel probes) or tracepoints. Whenever the kernel executes a function associated with that event, the bpftrace script fires.

In the Linux networking stack, a very common function responsible for discarding packets is kfree_skb (Free Socket Buffer). By monitoring every time the kernel calls kfree_skb, we can track packet drops.

Step 2: Writing a Basic Drop Trace Script

You can run bpftrace directly from the command line for quick diagnostics. To simply count how many times kfree_skb is called per second, use the following one-liner:

sudo bpftrace -e 'kprobe:kfree_skb { @drops = count(); } interval:s:1 { print(@drops); clear(@drops); }'

While useful for baseline metrics, this doesn’t tell us why the drop happened. For that, we need the kernel stack trace.

Step 3: Capturing the Kernel Stack Trace

To find the root cause, we need to see the exact sequence of kernel functions that led up to the kfree_skb call. We can modify our bpftrace script to print the kernel stack (kstack) whenever a drop occurs.

Create a file named dropwatch.bt:

# dropwatch.bt
kprobe:kfree_skb
{
    // Filter out normal packet freeing, focus only on actual error drops
    // The exact logic depends on the kernel version, but often drops
    // are passed through specific drop tracepoints.
}

tracepoint:skb:kfree_skb
{
    printf("Packet dropped! Kernel stack trace:\n%s\n", kstack);
}

Run the script:

sudo bpftrace dropwatch.bt

When a packet is dropped, your terminal will flood with the exact C-level function calls inside the Linux kernel. If you see functions like nf_hook_slow, the packet was dropped by an iptables firewall rule. If you see tcp_v4_rcv followed by tcp_v4_do_rcv and then a drop, the TCP socket buffer is likely full.

Step 4: Isolating Drops by IP Address

A global stack trace can be overwhelming on a busy server. eBPF allows us to dig into the actual packet data (the sk_buff structure) and filter our traces by source or destination IP.

Writing a bpftrace script to parse the IP header requires deep knowledge of kernel structs, but utilizing the bpfcc-tools package provides a pre-built utility called dropwatch or tcpdrop.

To use the pre-built BCC tool for TCP packet drops:

sudo /usr/share/bcc/tools/tcpdrop

This command automatically parses the kernel structures and cleanly outputs the Source IP, Destination IP, TCP state, and the kernel stack trace that caused the drop, completely eliminating the guesswork from network troubleshooting.

Conclusion

Network packet drops are no longer a black box. By leveraging the power of eBPF and the bpftrace utility, Linux administrators can surgically pinpoint exactly where and why the kernel is discarding traffic, allowing for rapid and definitive resolution of complex network bottlenecks.