The Hidden Storage Limit
A common nightmare scenario for Linux system administrators is receiving a “No space left on device” error when attempting to create a new file, even when the df -h command clearly shows that the hard drive has 500GB of free space available. How can a drive be simultaneously empty and completely full?
The answer lies in the filesystem architecture. In Linux (specifically on ext4 filesystems), storage is divided into two distinct limits: Blocks (which hold the actual physical data, measured in Gigabytes) and Inodes.
An inode is a data structure that stores the metadata (permissions, ownership, timestamps, and physical disk locations) for a single file or directory. When a filesystem is formatted, a fixed number of inodes are created. If your web server contains a poorly configured PHP script that generates millions of tiny 1-kilobyte cache files, you will exhaust your supply of inodes long before you exhaust your physical Gigabyte storage.
Once you run out of inodes, the filesystem is effectively full. It is mathematically impossible to create a new file.
Checking Inode Usage
To diagnose this issue, you must append the -i (inode) flag to the standard Disk Free (df) command.
Open your terminal and run:
df -i
The output will be a table that looks remarkably similar to the standard storage table, but it is measuring file counts, not file sizes.
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 1002345 456 1001889 1% /dev
tmpfs 1008765 789 1007976 1% /run
/dev/sda1 6553600 6553600 0 100% /
/dev/sdb1 15000000 45000 14955000 1% /mnt/data
Analyzing the Output
Look at the /dev/sda1 root partition. The IUse% column is sitting at exactly 100%. The system was allocated 6,553,600 total inodes during formatting, and all 6,553,600 are currently in use. This server is completely crippled and requires immediate intervention.
Hunting Down the Culprit
Now that you know you have an inode exhaustion problem, you must find out exactly which directory is hoarding millions of files. Because standard tools like du -sh calculate file size rather than file count, they are useless here.
You must use a custom combination of the find, wc (word count), and sort commands to tally the number of files in every top-level directory.
Run the following command as root:
for i in /*; do echo $i; find $i | wc -l; done
This script will slowly crawl through the root directories (/var, /usr, /home, etc.) and output the exact number of files contained within them.
/home
1450
/var
6540000
/usr
11500
The output clearly indicates that the /var directory is holding 6.5 million files. You would then repeat the script, drilling deeper into /var/*, then /var/lib/*, until you isolate the exact malfunctioning folder containing the millions of cache files. Deleting those files will instantly free the inodes and restore the server to full functionality.