Diagnosing network connectivity bottlenecks, unpredictable packet loss, and high latency is a routine responsibility for system administrators, DevOps engineers, and technical workstation users. While standard ping tests confirm basic round-trip reachability to a remote server, they cannot reveal where along the transit route network degradation occurs. In Linux environments, two command-line utilities provide essential visibility into network hops: traceroute and mtr (My Traceroute). Understanding how to use these tools enables you to pinpoint router failures, misconfigured autonomous systems (AS), and ISP peering congestion with surgical accuracy.
Understanding Packet Routing and TTL Mechanisms
Both traceroute and mtr rely on the Time-to-Live (TTL) field located inside IP packet headers:
- The TTL Decrement Rule: Every router or layer-3 switch along an internet path decrements the packet’s TTL value by 1 before forwarding it to the next hop.
- ICMP Time Exceeded Notifications: When a packet arrives at an intermediate gateway with a TTL of 1, the router discards the packet and transmits an
ICMP Time-to-Live Exceeded in Transit(Type 11, Code 0) message back to your originating Linux host. - Hop Discovery: By deliberately transmitting packets with incrementally increasing TTL values (starting at TTL=1, then TTL=2, TTL=3, and so on), diagnostic tools systematically reveal the IP address, hostname, and round-trip time (RTT) for every consecutive gateway between your host and the final destination.
Installing traceroute and mtr on Linux
While many server distributions include basic network diagnostic binaries, neither utility is guaranteed to be pre-installed out of the box. Install them via your package manager:
On Ubuntu, Debian, and Linux Mint:
sudo apt update
sudo apt install traceroute mtr -yOn Fedora, RHEL, and AlmaLinux:
sudo dnf install traceroute mtr -yOn Arch Linux and Manjaro:
sudo pacman -S traceroute mtrUsing traceroute to Map Routing Hops
By default on Linux, modern implementations of traceroute send outbound UDP datagrams over high-numbered ports (starting at port 33434) rather than ICMP echo requests.
1. Running a Standard Traceroute
To trace the transit path to a remote host:
traceroute example.comThe output displays numbered hops, reverse DNS hostnames, IP addresses, and three round-trip timing measurements (in milliseconds) representing three probe packets dispatched per hop.
2. Using ICMP Echo Probes
Many modern enterprise edge firewalls and cloud security groups block high-numbered UDP packets, resulting in lines of asterisks (* * *). To bypass this restriction, force ICMP echo packets (equivalent to the Windows tracert behaviour):
sudo traceroute -I example.com3. Using TCP SYN Probes
If intermediate network appliances drop both UDP and ICMP traffic, you can dispatch TCP SYN packets to standard web server ports (such as port 443 for HTTPS):
sudo traceroute -T -p 443 example.com4. Disabling Reverse DNS Resolution for Speed
If DNS lookups delay your diagnostics, pass the -n flag to output raw numeric IP addresses immediately:
traceroute -n example.comUsing mtr for Real-Time and Statistical Analysis
While traceroute provides a static one-time snapshot of network routing, mtr combines the functionality of traceroute and ping into a continuous, real-time diagnostic dashboard. It constantly refreshes network statistics, allowing you to observe transient packet loss spikes and jitter over minutes or hours.
1. Launching the Interactive Terminal Interface
To start real-time monitoring:
mtr example.comIn this dynamic ncurses interface, press d to toggle between packet display modes, n to toggle DNS resolution on and off, or q to quit.
2. Generating a Clean Report with the -r Flag
When documenting latency issues for hosting providers or ISPs, an automated report is significantly more useful than a live screen. The --report (or -r) flag sends a predetermined number of cycles (default 10) and prints a tabular summary before exiting:
mtr --report --report-cycles 20 -n example.comUnderstanding mtr Diagnostic Columns
The statistical output generated by mtr -r includes several metrics:
- Loss%: The percentage of probe packets that failed to return an ICMP response for that specific hop.
- Snt: The total number of probe packets transmitted to this hop.
- Last: The round-trip latency of the most recent probe packet.
- Avg: The mean round-trip latency across all transmitted probes.
- Best: The shortest round-trip time recorded during the test session.
- Wrst: The highest (worst) latency spike observed.
- StDev: The standard deviation of the latency values. A high standard deviation indicates severe packet jitter.
traceroute vs mtr Feature Comparison
| Feature | traceroute | mtr (My Traceroute) |
|---|---|---|
| Execution Model | Single sequential pass | Continuous live updates or batch report |
| Default Protocol (Linux) | UDP (ports 33434+) | ICMP Echo requests |
| TCP SYN Probing | Yes (-T -p <port>) |
Yes (--tcp -P <port>) |
| Jitter / Deviation Metrics | No | Yes (Standard Deviation column) |
| Packet Loss Calculation | Manual inspection of asterisks | Exact percentage per hop |
| Privilege Requirements | Unprivileged for UDP; sudo for ICMP/TCP | Requires raw socket access (sudo or setuid) |
How to Accurately Interpret Packet Loss and Router Rate Limiting
A frequent mistake when interpreting traceroute or mtr output is assuming that a middle hop with 50% packet loss represents a broken network link. Modern core internet routers prioritize routing actual transit customer traffic over responding to low-priority ICMP diagnostic messages:
- ICMP Control Plane Rate Limiting: If Hop 5 displays 60% loss, but Hop 6, Hop 7, and the final destination show 0% loss, there is no actual packet loss occurring on your traffic stream. Hop 5 is simply rate-limiting ICMP responses on its management processor.
- Genuine Network Degradation: If packet loss begins at Hop 5 and persists across all subsequent hops (Hop 6, Hop 7, and Destination), that router or peering link is suffering genuine congestion or hardware failure.
Frequently Asked Questions
Why do I see asterisks (* * *) for certain hops?
Asterisks indicate that the router did not return an ICMP Time Exceeded packet within the designated timeout window. This is typically due to firewall policies discarding UDP/ICMP probes, or routers configured not to generate ICMP error messages.
Can I test IPv6 networks with traceroute and mtr?
Yes. Add the -6 flag to force IPv6 resolution: traceroute -6 example.com or mtr -6 example.com. Alternatively, use the dedicated traceroute6 utility.
Is it possible to export mtr results as structured JSON?
Yes. You can output automated JSON metrics for log ingestion or alerting pipelines by running: mtr --json example.com.