The “Everything is a File” Philosophy
To troubleshoot a Linux system effectively, you must understand its core philosophy: everything is a file. A text document is a file. A directory is a file. A hardware device (like a hard drive or a webcam) is a file in /dev/. Even an active network connection (a TCP socket) is treated by the kernel as a file descriptor.
When an application behaves erratically—such as a web server failing to bind to port 80, or a backup script refusing to unmount a USB drive because it says “device is busy”—the root cause is almost always an invisible process holding a file descriptor open.
To instantly see exactly which process is interacting with which file or network socket, Linux administrators rely on the lsof (List Open Files) command. It is the ultimate diagnostic tool for unmasking hidden system dependencies.
Step 1: Identifying What is Blocking a Drive Unmount
A classic sysadmin headache is attempting to unmount a filesystem (like /mnt/usb) and receiving the error: umount: target is busy. This means some process is actively reading or writing to that directory.
Instead of blindly guessing or forcefully rebooting the server, use lsof to target the mount point:
sudo lsof +D /mnt/usb- The
+Dflag tellslsofto search the directory and all of its subdirectories recursively.
The output will clearly list the Command (e.g., bash or rsync), the PID (Process ID), and the specific User who is keeping the filesystem active. You can then gracefully close the application or use kill -9 [PID] to terminate the offending process and successfully unmount the drive.
Step 2: Finding the Process Listening on a Network Port
Because Linux treats network sockets as files, lsof is an exceptionally powerful network diagnostic tool, often replacing netstat or ss for specific queries.
If you attempt to start Nginx and it fails with Address already in use, it means another process has bound to TCP port 80 or 443. To find out exactly which application is stealing the port, use the -i (internet) flag:
sudo lsof -i :80The output will reveal the culprit—perhaps an old Apache instance (httpd) or a rogue Python script. You will see the exact PID and the user running it.
You can also use the -i flag to monitor specific protocols or states. For example, to list every established SSH connection on the server:
sudo lsof -i tcp:22 -s tcp:ESTABLISHEDStep 3: Tracking Down Deleted Files Consuming Disk Space
A bizarre but common Linux phenomenon is when the df -h command shows the disk is 100% full, but running du -sh /* shows you are only using 20% of the disk. Where did the space go?
In Linux, if you delete a massive log file (e.g., rm /var/log/syslog.1) while a process (like rsyslogd) is still actively writing to it, the file is removed from the directory tree, but the kernel will not release the disk blocks until the process closes the file descriptor.
You can use lsof to hunt down these “deleted but active” phantom files:
sudo lsof | grep deletedThis command pipes the entire active file list into grep, filtering for the (deleted) state. The output will show you exactly which process is holding the massive file open. Restarting that specific service (e.g., systemctl restart rsyslog) will release the file descriptor and instantly free up the disk space.
Step 4: Monitoring a Specific User or Process
If you suspect a specific application or user is misbehaving, you can filter lsof to show only their activity.
To see every file and network socket opened by the user www-data:
sudo lsof -u www-dataTo see every file opened by a specific process ID (e.g., PID 1234):
sudo lsof -p 1234This is incredibly useful for security auditing. If you see a suspicious PHP process running under www-data, checking its open files via -p might reveal it communicating with an external malicious IP address or attempting to read /etc/shadow.
Conclusion
Because the Linux kernel abstractions rely on the “everything is a file” model, the lsof command acts as an x-ray machine for the entire operating system. By mastering its flags, administrators can instantly resolve blocked unmounts, discover port conflicts, recover lost disk space, and perform deep forensic analysis on running processes.