Administering a Linux server or development machine requires routine filesystem writes—such as generating log files, installing packages with apt, or writing temporary session files. However, executing a basic write command only to be blocked by the error: “No space left on device” immediately suggests a full storage volume. Yet, running df -h reveals a perplexing contradiction: your primary partition shows 50 GB, 100 GB, or even terabytes of available capacity. This frustrating paradox occurs when the filesystem runs out of inodes (index nodes) rather than physical disk blocks. In traditional UNIX/Linux filesystems such as ext4, ext3, and ext2, every file, directory, and symbolic link requires an inode to store metadata. When millions of microscopic files are created, the filesystem depletes its allocated inode table completely, refusing to create any new files regardless of how much raw disk space remains. This guide outlines how to diagnose inode exhaustion and safely clear runaway file caches in Linux.
Understanding Block Storage vs Inode Allocation in Linux
Standard Linux filesystems track storage across two separate structures:
- Data Blocks: Physical sectors that store the actual binary contents of files (monitored via
df -h). - Inodes: Fixed data structures created at filesystem format time that store file metadata, permissions, ownership, timestamps, and pointers to data blocks (monitored via
df -i).
Because the total number of inodes is fixed upon filesystem creation, a server that hosts millions of tiny files (such as PHP session tokens, mailer queues, Docker container layer metadata, or automated log fragments) can easily consume 100% of available inodes while using less than 10% of physical disk capacity.
Method 1: Verify Inode Exhaustion Using df -i
The first diagnostic step is checking the inode allocation table across all mounted filesystems.
- Open your Linux terminal.
- Execute the following command to display inode utilization:
df -i - Inspect the output columns: Inodes, IUsed, IFree, and IUse%.
- Look for any filesystem displaying 100% under the IUse% column (commonly mounted on
/or/var). - If
IUse%is at 100% whiledf -hshows plenty of free gigabytes, your server is suffering from inode exhaustion.
Method 2: Locate Directories with Excessive File Counts
Traversing the filesystem to count the number of file entries per directory pinpoints the exact application creating millions of files.
- Execute this shell one-liner to scan and count directory entries, sorted from highest to lowest:
sudo find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n | tail -20 - Allow the scan to complete. The output will display the top 20 directories containing the highest number of individual files.
- Common culprits include:
/var/lib/php/sessions/(abandoned PHP session files)/var/spool/postfix/maildrop/or/var/spool/clientmqueue/(stalled email notifications)/tmp/or/var/tmp/(orphaned application scratch files)/var/lib/docker/overlay2/(dangling Docker image layers)
Method 3: Safely Delete Millions of Small Files
Running standard commands like rm * in directories containing hundreds of thousands of files fails with the error “Argument list too long”. Using find with the -delete flag bypasses shell argument limitations.
- For stale PHP session files older than 7 days:
sudo find /var/lib/php/sessions/ -type f -name 'sess_*' -mtime +7 -delete - For temporary files in
/tmp:sudo find /tmp/ -type f -atime +10 -delete - Alternatively, stream file paths directly into
xargs:sudo find /path/to/runaway/folder -type f | xargs -r rm -f - Run
df -ito confirm that millions of inodes have been instantly freed.
Method 4: Clean Dangling Docker Images and Build Cache
Containerized workloads generate massive volumes of tiny layer manifests and volume metadata.
- Check Docker disk and inode usage:
docker system df - Prune all unused containers, dangling networks, and untagged images:
docker system prune -a --volumes -f - Purge the Docker build cache:
docker builder prune -a -f - This operation frequently recovers hundreds of thousands of exhausted inodes on container hosts.
Method 5: Release Inodes Held by Deleted Files in Active Processes
When a log file is deleted while an active daemon process still holds its file descriptor open, Linux cannot release the inode until the process terminates.
- Identify processes holding open handles to unlinked/deleted files:
sudo lsof +L1 - Inspect the output for large file handles or numerous deleted entries belonging to services like Nginx, Apache, or MySQL.
- Restart the corresponding daemon to release the file handles:
sudo systemctl restart nginx - Execute
df -iagain to verify that inode usage has dropped back to normal levels.
For more Linux administration tutorials and filesystem troubleshooting solutions, visit the Digitash Terminal section.