The Locked File Nightmare
One of the most frustrating errors a Linux system administrator encounters is the “Device or resource busy” warning. You attempt to unmount a USB drive or a secondary hard drive using the umount command, and the kernel aggressively denies the request.
The kernel refuses to unmount the drive because a background process, an invisible daemon, or a forgotten bash session currently has a file open on that drive. If the kernel were to force the unmount, that application would crash, potentially causing severe data corruption.
The problem is that Linux doesn’t tell you which process is holding the drive hostage. Attempting to blindly kill random services is dangerous and amateurish. To mathematically identify the exact process causing the blockage, UNIX engineers use the lsof (List Open Files) command. lsof peers directly into the kernel’s file descriptor tables, exposing every single file, network socket, and directory currently open by every running process on the system.
Step 1: Understanding the Concept of “Files”
To master lsof, you must understand a fundamental UNIX philosophy: Everything is a file.
When you run lsof without any flags, it doesn’t just list standard text documents. It lists:
- Regular files (
.txt,.log). - Directories (if a user
cd‘d into a folder, that folder is considered “open”). - Network connections (TCP and UDP sockets).
- Hardware devices (e.g.,
/dev/null). - Libraries (e.g.,
.soshared objects loaded into RAM).
If you simply run sudo lsof on a busy production server, the terminal will instantly vomit 50,000 lines of output. You must use flags to filter the data.
Step 2: Identifying the Process Blocking a Drive
Let’s solve the primary use case. You are trying to unmount a secondary data drive mounted at /mnt/database_backups, and it says “Device is busy.”
You use lsof and pass the exact directory path as the argument.
sudo lsof /mnt/database_backupsThe output will look like this:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
bash 1455 jdoe cwd DIR 8,1 4096 2 /mnt/database_backups
tar 1602 root 3w REG 8,1 10485760 15 /mnt/database_backups/archive.tarDecoding the Output:
COMMAND: The name of the program holding the file open. (In this case, abashshell and atarbackup process).PID(Process ID): The exact numerical identifier of the running process. This is the critical number you need to kill it.USER: The user account running the process.FD(File Descriptor): Shows how the file is being used.cwdmeans Current Working Directory (userjdoeliterally typedcd /mnt/database_backupsand left their terminal window open).3wmeans it is open for Writing.
You now know exactly why the drive won’t unmount. You can ask jdoe to close their terminal, and you can wait for the tar backup to finish.
Step 3: Finding Who is Using a Network Port
Because network sockets are files in UNIX, lsof is a spectacular network troubleshooting tool, often replacing the need for netstat or ss.
Suppose you are trying to start the Nginx web server, but it crashes immediately with the error: “Port 80 is already in use.” You need to know what rogue application has hijacked port 80.
You use the -i (Internet) flag, followed by a colon and the port number.
sudo lsof -i :80The output will instantly reveal the culprit:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
apache2 2045 root 4u IPv4 12345 0t0 TCP *:http (LISTEN)You now know that a legacy apache2 process (PID 2045) is running in the background and squatting on the port.
Step 4: Investigating a Specific User or Process
If you suspect a compromised user account is acting maliciously, you can use lsof to see exactly what files that specific user currently has open.
Use the -u (User) flag:
sudo lsof -u hacker_accountThis will reveal if they are reading sensitive configuration files or opening outbound network sockets.
Conversely, if you want to know every single file a specific program (like a database) is touching, you can filter by the Process ID using the -p (PID) flag. If your PostgreSQL database is PID 5542:
sudo lsof -p 5542This is invaluable for debugging applications that are throwing “File not found” errors, as you can see exactly where the application is trying (and failing) to look on the hard drive.
Step 5: The Aggressive Kill Workflow (Automating the Fix)
In a severe outage, you may not have time to politely ask users to log out of a blocked directory. You need to forcefully unmount the drive immediately.
You can use the -t (Terse) flag. This strips away the beautiful columns and only outputs the raw Process IDs (PIDs). This allows you to pipe the output directly into the kill command.
To forcefully terminate every single process that is currently touching the /mnt/database_backups directory:
sudo lsof -t /mnt/database_backups | xargs sudo kill -9Warning: This is the nuclear option. kill -9 does not allow applications to save their data. It violently terminates them. But the moment the command executes, the directory will be completely freed, and your umount command will succeed instantly.
Conclusion
Blindly guessing why a filesystem won’t unmount or a network port won’t bind wastes critical time during an incident response. By mastering the lsof command, Linux administrators gain X-ray vision into the kernel’s internal state. The ability to mathematically link a locked file, a hijacked network port, or a hanging directory directly to the specific Process ID responsible provides the precise intelligence required to untangle complex system deadlocks.