Building resilient, fault-tolerant distributed systems and networked applications requires rigorous testing under adverse conditions. In local development or high-speed data centre environments, low-latency, error-free connections mask edge-case bugs such as timeout race conditions, retry storm cascading failures, and buffer bloat. The Linux kernel contains a built-in network emulation framework known as netem, managed via the tc (Traffic Control) subsystem. Without deploying third-party proxies or physical network attenuators, developers and systems engineers can inject precise packet delay, jitter distributions, packet loss, and corruption directly onto specific network interfaces.
Understanding Linux Traffic Control and Queuing Disciplines (qdisc)
In the Linux networking stack, outgoing packets traverse Queuing Disciplines (qdiscs) before transmission over physical or virtual network interface cards (NICs). The default qdisc is typically fq_codel or pfifo_fast. The netem qdisc replaces or augments standard queuing to artificially delay, drop, or reorder packets as they pass through the egress queue:
- Egress Traffic Control:
netemoperates strictly on outbound (egress) packets leaving an interface. To simulate degradation on incoming traffic, you must apply rules to the sender interface or route ingress traffic through an Intermediate Functional Block (IFB) pseudo-device. - Root Qdisc: When testing locally, rules are attached to the root of a specific interface (such as
eth0,wlan0, or virtual bridgebr0).
Injecting Deterministic Latency and Jitter
To simulate a high-latency Wide Area Network (WAN) connection, add the netem qdisc with an explicit delay parameter:
sudo tc qdisc add dev eth0 root netem delay 100ms
Every outbound packet on eth0 will now experience an exact 100-millisecond delay. In real-world cellular or satellite links, latency fluctuates unpredictably. You can introduce variance (jitter) and a normal distribution curve:
sudo tc qdisc change dev eth0 root netem delay 100ms 20ms distribution normal
In this configuration, outbound packet delays range between 80ms and 120ms (100ms ± 20ms), following a statistical normal distribution model that mirrors authentic mobile internet behaviour.
Simulating Packet Loss and Corruption
Network reliability testing frequently requires evaluating application resilience when segments of traffic disappear without warning:
- Random Packet Loss: To drop 5% of all outgoing packets at random:
sudo tc qdisc change dev eth0 root netem loss 5%
- Correlated (Burst) Loss: Real-world packet loss often occurs in clusters due to momentary physical interference or buffer exhaustion. You can define a correlation percentage (e.g., 5% loss with 25% correlation to previous packet state):
sudo tc qdisc change dev eth0 root netem loss 5% 25%
- Packet Corruption: To simulate line noise or faulty hardware by flipping single bits within packet payloads:
sudo tc qdisc change dev eth0 root netem corrupt 0.2%
Simulating Packet Duplication and Reordering
Certain network protocols, particularly UDP-based streaming and voice applications, degrade when packets arrive out of order or duplicated:
sudo tc qdisc change dev eth0 root netem duplicate 1%
To introduce out-of-order delivery, combine a delay baseline with a reordering percentage:
sudo tc qdisc change dev eth0 root netem delay 20ms reorder 25% 50%
Under this rule, 25% of packets are sent immediately while the remaining 75% are delayed by 20ms, forcing a predictable sequence gap.
Verifying and Deleting Traffic Control Rules
To inspect active traffic control qdiscs and verify current operational parameters:
tc qdisc show dev eth0
The terminal outputs the active qdisc parameters, confirming the configured delay, loss, and jitter rates.
Once testing is complete, restore the network interface to its default, uninhibited operating state by deleting the root qdisc:
sudo tc qdisc del dev eth0 root
The network interface immediately reverts to its native queuing discipline, restoring standard gigabit transmission without requiring an interface restart or system reboot.