How to Trace TCP Connect Failure Due to iptables and nftables Drops in Linux Using eBPF

Debugging outbound connection failures on Linux hosts often turns into an arduous trial-and-error process when firewall filter chains drop packets silently. While host services may report generic EPERM (Operation not permitted) or silent connection hangs, determining which specific Netfilter hook, table, or rule dropped the outbound SYN packet requires deep kernel inspection. Using eBPF-based Netfilter tracing tools, network administrators can trace packets as they traverse Netfilter chains and identify the exact rule causing connection drops in real time.

Netfilter Architecture: Hook Points and Packet Traversal

Both legacy iptables and modern nftables rely on the underlying Linux kernel Netfilter framework:

• NF_INET_LOCAL_OUT: The Netfilter hook point where locally generated outbound TCP packets are inspected before route lookup and transmission.
• NF_DROP (Verdict 0): Netfilter discards the packet without generating an ICMP unreachable error or notifying the sending socket immediately.
• NF_ACCEPT (Verdict 1): The packet passes inspection and continues down the network stack.
• eBPF Tracing Capability: By hooking into kernel functions such as nf_hook_slow(), eBPF captures the exact verdict, table name, hook number, and IP header information without modifying firewall rule sets.

Tracing Netfilter Drops with bpftrace

Deploy bpftrace to intercept the kernel’s internal Netfilter evaluation function nf_hook_slow, inspecting the return verdict for dropped outbound TCP packets:

sudo bpftrace -e '
#include <linux/skbuff.h>
#include <linux/ip.h>
#include <linux/netfilter.h>

kretprobe:nf_hook_slow
{
    // Verdict 0 represents NF_DROP
    if (retval == 0) {
        printf("FIREWALL DROP: Comm: %s (PID %d) | Return Verdict: %d (NF_DROP)
", 
               comm, pid, retval);
    }
}'

When an outbound connection triggers a firewall drop, bpftrace logs the calling process immediately:

FIREWALL DROP: Comm: curl (PID 41088) | Return Verdict: 0 (NF_DROP)
FIREWALL DROP: Comm: redis-cli (PID 41102) | Return Verdict: 0 (NF_DROP)

Pinpointing the Exact Dropping Rule with pwru (Packet, Where Are You?)

The open-source Cilium eBPF tool pwru traces packets throughout the entire Linux kernel networking stack, reporting the exact kernel functions that return error codes or drop packets:

# Install pwru binary
curl -sLO https://github.com/cilium/pwru/releases/latest/download/pwru-linux-amd64.tar.gz
tar -xzf pwru-linux-amd64.tar.gz
sudo mv pwru /usr/local/bin/

# Trace outbound drops to a specific destination IP
sudo pwru --filter-dst-ip 198.51.100.25 --output-tuple

When an outbound packet is blocked, pwru pinpoints the exact kernel location:

SKB: 0xffff8881023a1a00 | 10.0.1.5:51240 -> 198.51.100.25:80 (TCP)
  -> ip_output
  -> nf_hook_slow (Verdict: NF_DROP)
  -> kfree_skb (Reason: SKB_DROP_REASON_NETFILTER_DROP)

Correlating Drops with iptables and nftables Counters

Once eBPF confirms Netfilter is dropping the packets, locate the responsible rule by inspecting packet counters:

1. For nftables, display active rules with rule handles and packet hit counts:

   sudo nft -a list table inet filter

2. For legacy iptables, view packet counters across the OUTPUT chain:

   sudo iptables -L OUTPUT -v -n --line-numbers

3. To pinpoint transient drops dynamically, attach an explicit trace rule in nftables:

   sudo nft add rule inet filter output meta nftrace set 1
   sudo nft monitor trace

   The monitor stream outputs the table, chain, and rule handle that executed the drop verdict.