The Ultimate Network Truth
When troubleshooting complex network issues on a Linux server, high-level tools like ping or curl only tell you if a connection succeeded or failed. If a web server is inexplicably dropping connections from a specific subnet, or if you suspect a compromised application is exfiltrating plaintext data to an unknown IP address, application logs will not help you.
To understand exactly what is happening on the network, you must inspect the raw electrical traffic passing through the Network Interface Card (NIC). You must look at the actual packets.
The tcpdump command is the industry standard for command-line packet sniffing and deep packet inspection. It intercepts raw Ethernet frames, strips off the headers, and displays the exact contents of the network traffic in real-time. It is the command-line equivalent of Wireshark, allowing Linux administrators to definitively prove what data is entering and leaving their servers.
Step 1: Installing and Basic Sniffing
tcpdump requires root privileges to hook into the network stack. It is usually installed by default, but if not:
sudo apt update
sudo apt install tcpdump -yTo initiate a basic sniff, you must tell tcpdump which network interface to listen on using the -i flag. (You can find your active interface, e.g., eth0 or ens33, by running ip addr).
sudo tcpdump -i eth0The terminal will instantly flood with hundreds of lines of data, representing every single packet hitting the server. To stop the flood, press Ctrl+C. tcpdump will print a summary showing how many packets it captured and how many were dropped by the kernel.
Step 2: Filtering by IP Address and Port
A raw tcpdump on a busy production server is completely unreadable. You must aggressively filter the traffic using BPF (Berkeley Packet Filter) syntax.
If you are trying to debug a connection issue between your server and a specific database located at 10.50.1.100, you filter by the host keyword:
sudo tcpdump -i eth0 host 10.50.1.100This will only display packets where 10.50.1.100 is either the source or the destination.
If you want to monitor all HTTP traffic hitting your web server, regardless of the IP address, you filter by the port keyword:
sudo tcpdump -i eth0 port 80You can chain filters together using logical operators (and, or, not). To see all traffic to port 80, except the traffic coming from your own management IP (192.168.1.50):
sudo tcpdump -i eth0 port 80 and not src 192.168.1.50Step 3: Controlling the Output Format (-n and -v)
By default, tcpdump tries to be helpful by performing reverse DNS lookups on every IP address it sees, attempting to translate 8.8.8.8 into dns.google.com. This is incredibly slow and will cause tcpdump to drop packets on a busy network.
Always use the -n flag. This forces tcpdump to print raw IP addresses and raw port numbers, bypassing DNS resolution entirely.
sudo tcpdump -i eth0 -n port 443If you need more details about the packet (such as the TTL, the IP ID, or TCP sequence numbers for debugging dropped connections), you add the -v (verbose) flag. You can stack it (-vv or -vvv) for extreme detail.
sudo tcpdump -i eth0 -n -vv host 10.50.1.100Step 4: Inspecting the Payload (ASCII and Hex)
By default, tcpdump only prints the packet headers (Source IP, Dest IP, Port, Flags). It does not print the actual data (the payload) contained inside the packet.
If you suspect an application is sending passwords in plaintext over port 21 (FTP) or port 80 (HTTP), you can instruct tcpdump to print the raw payload using the -X (Hex and ASCII) or -A (ASCII only) flags.
sudo tcpdump -i eth0 -n -A port 80If you run this command and log into a non-HTTPS website, you will literally see your username=admin&password=secret string print across the terminal in plaintext. This definitively proves to developers why unencrypted protocols must be banned.
Step 5: Capturing to a PCAP File for Wireshark
Reading complex TCP handshakes (SYN, SYN-ACK, ACK) scrolling by in a terminal is difficult even for seasoned engineers. The ultimate workflow is to capture the packets on the headless Linux server, save them to a file, and then download that file to your laptop to analyze it using the graphical Wireshark interface.
To save the capture to a file instead of printing it to the screen, use the -w (write) flag. You must also specify the -s 0 (snaplength) flag to ensure tcpdump captures the entire packet, not just the first 68 bytes.
sudo tcpdump -i eth0 -n -s 0 port 3306 -w database_trace.pcapLet this run until the issue occurs, then press Ctrl+C. You now have a perfectly formatted .pcap file. You can download this file via scp and open it in Wireshark, where you can graphically filter streams, reconstruct full TCP conversations, and effortlessly pinpoint the exact packet that caused the connection reset.
Conclusion
Logs can lie, and applications can crash silently, but packets on the wire represent the absolute, irrefutable truth of network communications. By mastering tcpdump, BPF filtering syntax, and PCAP file generation, Linux administrators gain the ultimate diagnostic capability, allowing them to surgically intercept and analyze any data stream entering or leaving their infrastructure.