The Blindness of the Ping Command
When an Ubuntu server experiences agonizingly slow connections to an external API (like an AWS database), the standard instinct of a junior administrator is to run the ping command. If ping 8.8.8.8 returns a stable 15-millisecond response time, the administrator erroneously concludes the network is flawless and blames the application.
This is a catastrophic diagnostic failure. The ping command only measures the aggregate round-trip time to the final destination. It is entirely blind to the chaotic journey those packets take. A packet traveling from your Ubuntu server in New York to an AWS server in London might traverse 15 different physical routers owned by 5 different telecommunication companies. If Router #7 (owned by a transit provider in the Atlantic Ocean) is suffering a CPU failure and dropping 40% of its packets, ping will not tell you where the failure is occurring.
Furthermore, traceroute is equally insufficient. It maps the path once, but it does not provide sustained, statistical evidence of packet loss over time. To mathematically map the exact topological route of a packet and continuously interrogate the latency and stability of every single physical router along that path, network engineers deploy the mtr (My Traceroute) utility.
Step 1: The Architecture of the Time-To-Live (TTL) Hack
Before running mtr, you must understand how it mathematically forces routers to reveal themselves.
Every IP packet has a Time-To-Live (TTL) integer in its header. Every time a router processes the packet, it subtracts 1 from the TTL. If the TTL hits 0, the router destroys the packet and sends an ICMP “Time Exceeded” error back to the original sender.
mtr weaponizes this mechanism. It fires a packet with a TTL of 1. The very first router (your default gateway) drops it and replies. mtr logs the IP address and latency. It then fires a packet with a TTL of 2. The second router drops it and replies. mtr repeats this process hundreds of times per second, continuously forcing every single router in the chain to mathematically identify itself and prove its processing speed.
Step 2: Executing the Continuous Interrogation
Install the mtr package on your Ubuntu server:
sudo apt update
sudo apt install mtr -yExecute the command against the problematic destination (e.g., Google’s public DNS):
mtr 8.8.8.8The terminal transforms into a live, continuously updating dashboard. It will list the “Hops” (the routers) sequentially from 1 to 15. The dashboard provides critical telemetry:
- Loss%: The exact percentage of packets that the specific router failed to return.
- Snt (Sent): The number of probe packets fired.
- Last, Avg, Best, Wrst: The statistical latency calculation (in milliseconds).
Step 3: Diagnosing Control Plane vs. Data Plane Drops
When analyzing mtr output, junior engineers frequently misinterpret the data, leading to false panic.
Suppose you see 15 hops. Hop 4 shows a terrifying 80% Packet Loss. However, Hop 5, Hop 6, and the final destination (Hop 15) all show 0% Packet Loss.
This is a Control Plane limit, not a network failure. Enterprise backbone routers are designed to route terabytes of data (the Data Plane) instantly. However, their CPUs (the Control Plane) are weak. When mtr forces the router to generate an ICMP “Time Exceeded” error, it taxes the CPU. To protect itself from Denial of Service (DoS) attacks, Hop 4 is intentionally dropping the ICMP replies. Because the traffic successfully reaches the final destination, Hop 4 is perfectly healthy; it is simply ignoring your probes.
The Rule of Thumb: You only have a true network problem if the packet loss starts at a specific hop and continues all the way to the final destination.
Step 4: Exposing Asymmetric Routing
The most difficult network anomaly to diagnose is Asymmetric Routing. This occurs when a packet takes Path A to reach London, but the reply packet takes Path B to return to New York. Path A might be a flawless fiber-optic cable, but Path B might be heavily congested.
If you suspect asymmetric routing, you must execute mtr simultaneously from both ends of the connection.
Open an SSH session to your Ubuntu server in New York and run:
mtr --report --report-cycles=100 <LONDON_IP>Simultaneously, open an SSH session to the Ubuntu server in London and run:
mtr --report --report-cycles=100 <NEW_YORK_IP>The --report flag commands mtr to run invisibly in the background, send exactly 100 packets, and dump a final, static mathematical summary to the terminal. You compare the two reports. If New York shows 15 hops going through a “Level3” transit provider, but London shows 22 hops going through a “Cogent” transit provider, you have mathematically proven the existence of a massive asymmetric route, which explains the unpredictable latency spikes your application is experiencing.
Step 5: Bypassing ICMP Firewalls (TCP Probing)
Many modern cloud providers (including AWS and Azure) aggressively block ICMP (Ping) packets at their edge firewalls. If you run a standard mtr against an AWS server, it will hit Hop 12 and then display endless ??? because the firewall is dropping the probes.
To mathematically bypass the ICMP firewall, you force mtr to use TCP packets (specifically targeting an open port like 443 for HTTPS).
sudo mtr --tcp --port 443 8.8.8.8The AWS firewall perceives this as legitimate web traffic and allows the packets to penetrate the perimeter. The target server responds with a standard TCP RST (Reset), allowing mtr to accurately map the internal cloud topology that was completely invisible to standard diagnostic tools.
Conclusion
Relying on basic connectivity tests like ping provides a dangerous, superficial view of enterprise network health. By mastering the mtr utility, network engineers weaponize the TTL architecture to violently expose the hidden infrastructure of the internet. The ability to mathematically isolate packet loss to specific transit providers, continuously monitor latency jitter, and bypass ICMP firewalls via TCP probing transforms vague connectivity complaints into precise, actionable network topology intelligence.