How to Use lsof in Linux to Trace Open File Descriptors, Network Sockets, and Deleted File Locks

One of the core philosophical tenets of the UNIX and Linux operating systems is that “everything is a file”. Beyond standard text documents and binary executables, the Linux Virtual File System (VFS) treats network TCP/UDP sockets, hardware storage blocks, IPC pipes, character devices, and kernel memory buffers as file descriptors. Consequently, when a Linux server suffers from mysterious disk exhaustion, port binding failures, or hung filesystem unmounts, inspecting open file handles is essential.

The definitive Linux command-line utility for system-wide file descriptor interrogation is lsof (List Open Files). By querying kernel data structures exposed through the /proc virtual filesystem, lsof allows systems engineers to pinpoint processes holding locks on unlinked deleted files (resolving the classic discrepancy where df shows a disk full while du shows gigabytes of free space), audit open network listening sockets, and prevent file descriptor exhaustion.

Architectural Overview: The Linux VFS and Process File Descriptor Tables

Every process running in Linux maintains a private file descriptor table mapped into kernel memory:

Layer	Technical Architecture	Operational Role
File Descriptor (FD)	Per-Process Integer Index (0, 1, 2… N)	Non-negative integer indexing a process’s open file table (0=stdin, 1=stdout, 2=stderr).
Kernel Open File Table	Global Inode & Offset Structure	Tracks current byte read/write offset, access modes (read/write/append), and active reference count.
Inode Table	Filesystem Metadata Object	Contains physical block pointers and link count (nlink). File blocks are freed only when nlink == 0 and active FD count reaches zero.
Socket Inode	Network Protocol Control Block	Maps network endpoints (IPv4/IPv6, TCP/UDP states) into the file descriptor architecture.

Identifying Unlinked Deleted Files Holding Disk Space

A frequent operational incident occurs when an automated script deletes a massive multi-gigabyte log file (e.g. rm -f /var/log/app.log), yet df -h reveals no disk space was reclaimed. In Linux, deleting a file merely decrements its filesystem directory link counter. If an active process (such as Nginx, Java, or PostgreSQL) still holds that file descriptor open, the kernel preserves the raw disk blocks until the process terminates.

Step 1: Locate Deleted Files with Open File Descriptors

Run lsof filtered for deleted markers:

# Identify all deleted files still held open in memory
sudo lsof +L1

Alternatively, grep specifically for unlinked descriptors:

sudo lsof | grep -i "(deleted)"

Step 2: Interpret the Diagnostic Output

The output reports the offending process and its resource consumption:

COMMAND   PID   USER   FD   TYPE   DEVICE     SIZE/OFF   NLINK   NODE   NAME
java     4821   app    14w   REG    259,2   42949672960      0   8921   /var/log/app.log (deleted)

• COMMAND / PID: The binary and Process ID keeping the handle alive (Java, PID 4821).
• SIZE/OFF: The unreleased disk space (e.g. ~40GB).
• NLINK 0: Proves the file was unlinked from the directory tree.
• FD 14w: File descriptor number 14, open in write mode (w).

Step 3: Reclaim Disk Space Without Killing the Process

If the service cannot be restarted immediately in production, truncate the open file descriptor directly through the /proc filesystem:

# Zero-out the unlinked file descriptor to free disk blocks instantly
sudo truncate -s 0 /proc/4821/fd/14

The kernel immediately reclaims the 40GB of flash blocks without crashing the Java application.

Auditing Network Sockets and Port Bindings

lsof excels at diagnosing port collisions (such as “Address already in use”) and identifying rogue network listening daemons.

Step 1: Identify Process Listening on a Specific Port

# Find the process bound to TCP port 80 or 443
sudo lsof -i :80
sudo lsof -i :443

Step 2: Filter by Protocol and State

List all active, established TCP connections across the server:

# List all established TCP sockets
sudo lsof -iTCP -sTCP:ESTABLISHED

Display all listening UDP services without resolving hostnames or port names to speed up command execution:

# Fast numerical network inspection (-n prevents DNS lookup, -P prevents port name mapping)
sudo lsof -nP -iUDP

Tracing Filesystem Locks Preventing Mount Detachments

When attempting to unmount an external SAN volume, NFS share, or USB drive via umount /mnt/backup, Linux frequently throws the error: umount: /mnt/backup: target is busy.

Use lsof with the recursive directory flag (+D) to find every process operating inside that directory tree:

# Identify all processes with open files or active working directories in /mnt/backup
sudo lsof +D /mnt/backup

If an administrator’s Bash shell has its current working directory (cwd) set to /mnt/backup, or a background worker is reading a file, lsof outputs the PID. Terminate the blocking process or change directories, then unmount cleanly.

Filtering by User, Process, and Combining Conditions with -a

By default, passing multiple options to lsof combines them using an OR condition. To apply strict AND filtering, pass the -a operator.

# Find open network sockets belonging strictly to user 'www-data' (AND logic)
sudo lsof -a -u www-data -i

# Inspect all open files and libraries held by a specific Process ID
sudo lsof -p 1042

Troubleshooting lsof Execution and Performance Overhead

Consult this diagnostic reference table to debug permission blocks and performance lags:

Issue	Underlying Root Cause	Remediation Procedure
lsof outputs only a handful of user processes.	Command executed without root/sudo privileges; unprivileged users cannot inspect other users’ file descriptors in /proc.	Always execute sudo lsof when troubleshooting system-wide or daemon-level resources.
lsof execution hangs or takes several minutes to complete.	Command is attempting reverse DNS lookups on every open socket and resolving network file shares (NFS).	Always append -n (disable DNS) and -P (disable port name resolution): sudo lsof -nP. Avoid +D on network mounts.
Server logs report Too many open files (EMFILE / ENFILE).	Process or kernel exceeded max descriptor limits (ulimit -n or fs.file-max).	Check active system limits via cat /proc/sys/fs/file-nr. Expand per-process limits in /etc/security/limits.conf (e.g. * soft nofile 65535).
lsof: WARNING: can't stat() fuse.sshfs file system.	A FUSE or SSHFS network mount has hung, preventing stat calls.	Use -b flag to avoid stat calls on hung filesystems: sudo lsof -b.

Mastering lsof file descriptor inspection, network socket tracking, and unlinked file recovery equips systems engineers with deep visibility into the Linux kernel, resolving complex storage leaks and port collisions with surgical precision.