How to Trace TCP PMTU Discovery Black Hole Drops in Linux Using eBPF

Path MTU Discovery (PMTUD) allows TCP endpoints to discover the maximum transmission unit size supported across an entire internet routing path without fragmenting packets. Senders transmit full-sized packets with the Don’t Fragment (DF) bit set. If an intermediate router cannot forward a 1500-byte packet across a constrained tunnel (e.g., PPPoE, IPSec, or VXLAN), it discards the packet and transmits an ICMP "Fragmentation Needed" message (Type 3, Code 4) back to the sender. When misconfigured firewalls drop these ICMP messages, a PMTU black hole occurs: small handshakes succeed, but large data transfers freeze indefinitely. Tracing these drops with eBPF isolates black hole stalls immediately.

Understanding PMTU Black Hole Failure Dynamics

The sequence leading to a PMTU black hole stall follows a distinct pathology:

• SYN Handshake Success: Initial SYN and ACK packets are typically small (under 100 bytes) and easily pass through intermediate tunnels without fragmentation.
• Large Payload Stall: When the server attempts to transmit a 1460-byte payload, the intermediate gateway discards the packet due to a 1400-byte MTU bottleneck.
• ICMP Filtering: The gateway issues an ICMP Type 3 Code 4 notification, but upstream network filters drop all ICMP packets as a misguided security measure.
• Sender Retransmission Loop: The sender never receives the MTU feedback, repeatedly retransmitting the exact same 1460-byte packet until connection timeout.

Auditing Active PMTU Probing Settings in Linux

Inspect existing kernel settings governing MTU discovery and black hole detection:

# Check PMTU Discovery status (1 = enabled)
sysctl net.ipv4.ip_no_pmtu_disc

# Check TCP MTU Probing (0=disabled, 1=prober on blackhole, 2=always probe)
sysctl net.ipv4.tcp_mtu_probing

# Inspect cached PMTU routes in the kernel metrics cache
ip tcp_metrics show | grep -i pmtu

Tracing ICMP Type 3 Notifications with bpftrace

Attach an eBPF tracepoint to monitor incoming ICMP error messages and verify whether the kernel receives MTU reduction signals:

sudo bpftrace -e '
#include <linux/icmp.h>
#include <linux/ip.h>

kprobe:icmp_unreach
{
    $skb = (struct sk_buff *)arg0;
    $iph = (struct iphdr *)($skb->head + $skb->network_header);
    $icmph = (struct icmphdr *)($skb->head + $skb->transport_header);

    $type = $icmph->type;
    $code = $icmph->code;

    if ($type == 3 && $code == 4) {
        $next_mtu = ntohs($icmph->un.frag.mtu);
        printf("ICMP Frag Needed Received: From %s -> Next Hop MTU: %u bytes\n",
               ntop($iph->saddr), $next_mtu);
        @icmp_frag_needed_count = count();
    }
}'

If large transfers freeze and this probe records zero events, an upstream router is blackholing ICMP messages.

Tracing tcp_mtu_probe Invocations with eBPF

When Linux detects consecutive retransmission timeouts without progress, it triggers PLPMTUD (Packetization Layer Path MTU Discovery) via tcp_mtu_probe if enabled:

sudo bpftrace -e '
#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_mtu_probe
{
    $sk = (struct sock *)arg0;
    $tp = (struct tcp_sock *)arg0;

    printf("PLPMTUD Black Hole Probe Sent: PID %-6d Comm %-12s MSS Probe: %u\n",
           pid, comm, $tp->mss_cache);
    @plpmtud_probe_events = count();
}'

Remediating PMTU Black Holes in Linux

Resolve black hole deadlocks without manual per-route clamping:

• Enable TCP MTU Probing: Instruct Linux to search for smaller operational MSS sizes when unacknowledged retransmissions accumulate:

  sudo sysctl -w net.ipv4.tcp_mtu_probing=1
  sudo sysctl -w net.ipv4.tcp_base_mss=1024

• Clamping MSS at the Firewall (TCPMSS): On edge routers running iptables or nftables, enforce MSS clamping to match tunnel MTUs automatically:

  sudo iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu