How to Trace TCP PMTU (Path MTU) Blackholes and Packet Fragmentation in Linux Using eBPF

In modern cloud and multi-tier networking environments, encapsulation protocols such as VXLAN, Geneve, GRE, and IPsec reduce the effective Maximum Transmission Unit (MTU) along the network transit path. To optimize throughput, Linux TCP sockets set the “Don’t Fragment” (DF) bit on outbound packets, relying on Path MTU Discovery (PMTUD) to negotiate payload sizes. When an intermediate router drops an oversized packet but security firewalls block the returning ICMP Type 3 Code 4 (“Fragmentation Needed”) packet, a PMTU Blackhole occurs. Connections establish normally during the small-packet three-way handshake, but freeze indefinitely as soon as full-size data packets transmit. Using eBPF, Linux administrators can trace PMTU discovery failures directly in the kernel.

Understanding Linux PMTU Discovery Mechanics

The Linux kernel manages MTU negotiation through internal route metrics and ICMP message handlers:

• IP_PMTUDISC_DO: Default socket policy. Sets the DF bit on all TCP segments and expects downstream intermediate hops to notify the sender if an interface MTU is exceeded.
• ICMP Destination Unreachable (Type 3, Code 4): The router specifies its next-hop MTU. The Linux kernel processes this via tcp_v4_err(), reducing the connection’s Maximum Segment Size (MSS) stored in the route cache.
• PMTU Blackhole Manifestation: When ICMP packets are filtered, the sender never learns of the smaller MTU. Sockets enter continuous retransmission loops, freezing bulk transfers while interactive shell sessions remain deceptively responsive.

Tracing ICMP Fragmentation Errors with bpftrace

Deploy a bpftrace one-liner to intercept incoming ICMP error messages destined for local TCP sockets:

sudo bpftrace -e '
#include <net/sock.h>
#include <linux/icmp.h>
#include <net/tcp.h>

kprobe:tcp_v4_err
{
    $skb = (struct sk_buff *)arg0;
    $type = arg1 >> 8;
    $code = arg1 & 0xFF;
    
    // Check if ICMP Type 3 (Destination Unreachable) and Code 4 (Frag Needed)
    if ($type == 3 && $code == 4) {
        printf("PMTU EVENT: ICMP Frag Needed received! Socket updated to lower MSS.
");
    }
}'

Tracing Silent Re-transmits of MTU-Sized Segments

If intermediate firewalls drop ICMP packets, the kernel will not invoke tcp_v4_err(). To detect blackholes, trace consecutive retransmissions that specifically exceed standard sub-MTU sizes (payloads > 1400 bytes):

sudo bpftrace -e '
#include <net/sock.h>
#include <net/tcp.h>

kprobe:tcp_retransmit_skb
{
    $sk = (struct sock *)arg0;
    $skb = (struct sk_buff *)arg1;
    $len = $skb->len;

    // Detect full-size MTU segment retransmissions (potential PMTU blackhole)
    if ($len > 1300) {
        $inet = (struct inet_sock *)$sk;
        printf("BLACKHOLE SUSPECT: Comm: %s | Re-transmitting large segment (len=%d) to %s:%d
",
               comm, $len, ntop($inet->inet_daddr), bswap($inet->inet_dport));
    }
}'

Configuring Kernel Mitigation and Path MTU Probing

To eliminate PMTU blackhole vulnerabilities without lowering physical interface MTUs across the fleet:

1. Enable TCP Path MTU Probing (RFC 4821):

   sudo sysctl -w net.ipv4.tcp_mtu_probing=1

   Setting tcp_mtu_probing=1 instructs the kernel to detect blackholes automatically by testing smaller MSS payloads when consecutive retransmissions occur on large segments, entirely bypassing the need for ICMP feedback.
2. Adjust Base MSS Clamping at the Firewall:

   In networks using iptables or nftables on router gateways, clamp MSS to the path MTU automatically:

   sudo iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

3. Persist Configuration in sysctl.d:

   echo "net.ipv4.tcp_mtu_probing = 1" | sudo tee /etc/sysctl.d/99-tcp-pmtu.conf
   sudo sysctl --system