In Unix and Linux operating systems, the foundational design tenet asserts that “everything is a file.” Beyond regular text files and binary images, network sockets, named pipes (FIFOs), block devices, character devices, and shared memory segments are all exposed to processes as open file descriptors. However, this architectural consistency introduces common administrative dilemmas: an administrator attempting to unmount an external storage volume receives the infamous target is busy error, or a web server fails to bind with Address already in use because an orphaned zombie process retains an open socket handle.
Diagnosing and remediating these lockups without resorting to disruptive system reboots requires precision tooling. The lsof (LiSt Open Files) and fuser (File USER) utilities inspect the kernel’s virtual filesystem (VFS) and /proc filesystem structures directly. While lsof provides extensive analytical detail regarding process ownership, file descriptor flags, and network protocol states, fuser delivers rapid file-to-process lookups with built-in signal dispatch capabilities for instant process termination. This guide provides an operational manual for leveraging lsof and fuser in Linux to trace locked files, discover rogue socket bindings, unblock busy mount points, and safely terminate blocking processes.
Architectural Overview: lsof vs fuser Kernel Inspection
Understanding how lsof and fuser query the Linux kernel ensures efficient diagnostic selection:
| Operational Metric | lsof (LiSt Open Files) |
fuser (File User) |
|---|---|---|
| Data Source | Parses /proc/<PID>/fd/, /proc/<PID>/maps, and kernel VFS tables |
Directly queries /proc/<PID>/ descriptor trees |
| Output Formatting | Detailed tabular format (Command, PID, User, FD, Type, Device, Size, Node, Name) | Compact list of PIDs annotated with single-letter access flags |
| Process Termination | Informational only; requires piping PIDs to kill or xargs |
Native kill mechanism via -k / --kill parameter |
| Network Socket Inspection | Comprehensive (IPv4, IPv6, TCP state, UDP, raw ports) | Simple TCP/UDP port mapping via -n tcp <port> syntax |
| Primary Use Case | Deep forensic auditing, open file analytics, complex multi-criteria filtering | Rapid mount point clearance, socket eviction, and automated recovery scripting |
Prerequisites and Environment Verification
- Operating System: Any Linux distribution (Ubuntu, Debian, Red Hat Enterprise Linux, Rocky Linux, Fedora, Alpine).
- Privilege Level: Root or
sudoaccess is mandatory. While non-privileged users can execute both utilities, they can only view file descriptors belonging to their own user session; inspecting system daemons or other users requires root. - Package Availability: Both utilities are available in core package repositories:
Note: The# Debian / Ubuntu sudo apt update && sudo apt install -y lsof psmisc # RHEL / Rocky Linux / Fedora sudo dnf install -y lsof psmiscfusercommand is packaged inside thepsmiscsuite.
Step 1: Identifying Open Files and Directories with lsof
Executing lsof without arguments prints every open file descriptor across the entire operating system (often exceeding 50,000 lines). Applying focused search filters is essential:
Discovering Which Process is Accessing a Specific File
sudo lsof /var/log/nginx/access.log
The output reports the exact COMMAND (e.g., nginx), PID, USER (e.g., www-data), and file descriptor access mode (e.g., 1w for write access).
Recursively Inspecting an Entire Directory Tree
When attempting to modify, move, or archive an application directory:
# The +D flag inspects the directory and all nested subdirectories
sudo lsof +D /var/www/html/
Filtering Open Files by User or Process Name
# Find all files opened by user 'postgres'
sudo lsof -u postgres
# Find all files opened by the 'mysqld' daemon
sudo lsof -c mysqld
Step 2: Resolving “Target is Busy” Mount Conflicts
When unmounting an NFS share, USB storage device, or internal disk partition (e.g., umount /mnt/backup) fails with device is busy, running processes are holding open directory handles or locks on the mount point.
Method A: Auditing with lsof
# Inspect all open files on the mount point
sudo lsof +f -- /mnt/backup
Method B: Evicting and Killing with fuser (Fastest)
fuser specializes in resolving mount point deadlocks:
# 1. Enumerate all PIDs holding open handles on the filesystem mount
sudo fuser -v -m /mnt/backup
# 2. Interactively prompt to kill each blocking process
sudo fuser -k -i -m /mnt/backup
# 3. Forcefully terminate all blocking processes with SIGKILL
sudo fuser -k -9 -m /mnt/backup
# 4. Safely unmount the freed device
sudo umount /mnt/backup
Understanding fuser Access Code Letters
When running fuser -v, single-letter flags describe how each process interacts with the target:
c: Process is using the directory as its current working directory (CWD).e: Executable binary is currently running from this file path.f: Open file handle (default mode).r: Root directory (process has chrooted to this location).m: Memory-mapped file or shared library (mmap).
Step 3: Diagnosing and Terminating Locked Network Ports
When starting an application returns listen tcp :8080: bind: address already in use, you can immediately identify the blocking process using either tool:
Auditing Network Sockets with lsof
# Find process listening on TCP port 8080
sudo lsof -i TCP:8080 -sTCP:LISTEN
# Find all active IPv4 and IPv6 network connections
sudo lsof -i
# Find processes communicating with a specific remote host
sudo lsof -i @192.168.1.100
To terminate the offending process using lsof output:
sudo kill -9 $(sudo lsof -t -i :8080)
Note: The -t flag outputs raw PIDs exclusively, perfectly formatted for shell piping.
Auditing and Killing Sockets with fuser
# Identify process listening on TCP port 8080
sudo fuser 8080/tcp
# Instantly terminate the process holding TCP port 8080
sudo fuser -k 8080/tcp
Step 4: Recovering “Deleted” Files Held Open by Active Processes
In Linux, if a log file or database is deleted via rm while a process continues writing to it, the disk space is not reclaimed because the inode remains active until the file descriptor closes. lsof allows you to recover the orphaned data:
- Find unlinked files still consuming disk space:
Look for entries markedsudo lsof +L1(deleted). Note thePIDand file descriptor number (e.g., PID4192, FD3). - The Linux kernel maintains a link to the data in the
/procvirtual filesystem. - Copy the file data back to a safe location before killing the process:
sudo cp /proc/4192/fd/3 /tmp/recovered_data.log - Terminate or restart the daemon to permanently release the disk blocks.
Comparison of Common Operations
| Operational Goal | lsof Command |
fuser Command |
|---|---|---|
| Inspect File | lsof /path/to/file |
fuser /path/to/file |
| Inspect Mount Point | lsof +f -- /mnt/dir |
fuser -m /mnt/dir |
| Inspect TCP Port | lsof -i :443 |
fuser 443/tcp |
| Kill Process on Port | kill $(lsof -t -i :443) |
fuser -k 443/tcp |
| Kill Processes on Mount | kill $(lsof -t +f -- /mnt/dir) |
fuser -k -m /mnt/dir |
| Verbose Output | lsof (Default) |
fuser -v |
Troubleshooting Common Execution Errors
Issue 1: lsof Output Hangs Indefinitely
Root Cause: By default, lsof attempts to resolve network IP addresses into hostnames using reverse DNS and queries active NFS mounts. If an NFS server is unresponsive or DNS is misconfigured, lsof blocks inside kernel socket read calls.
Remediation: Execute with -n (disable host DNS lookups), -P (disable port name mapping), and -b (avoid blocking kernel calls):
sudo lsof -n -P -b
Issue 2: fuser Returns Zero Output Despite Port Being Busy
Root Cause: The port is occupied by a Linux container (Docker/Podman) running inside a separate network namespace, or fuser was executed without root privileges.
Remediation: Always run with sudo. If containers are active, query host bridge mappings using sudo ss -tulpn or inspect container socket bindings with docker ps.
Issue 3: “Device busy” Persists Even After fuser -k -9 -m
Root Cause: A process is in an uninterruptible sleep state (D state), typically waiting on dead hardware I/O or an orphaned kernel NFS RPC call. SIGKILL cannot terminate processes in D state.
Remediation: Identify D-state tasks with ps aux | awk '$8 ~ /D/'. If dealing with an unresponsive network mount, perform a lazy unmount: sudo umount -l /mnt/backup, which detaches the filesystem immediately from the hierarchy while cleaning up descriptors in the background.