Inode Exhaustion: The Disk Full Error That Makes No Sense

You run df -h and the root partition shows 85% used. Plenty of space left, right? Then you try to create a new file and get: No space left on device. You check again. You reboot. You even resize the partition. Same error. The problem isn't your disk space. It's your inodes. What Are Inodes? An inode is a data structure that stores metadata about a file: permissions, owner, timestamps, and pointers to the actual data blocks on disk. Every file, every directory, every symlink gets one inode. When you create a file, the system needs two things: Data blocks to store the actual content (governed by disk space) An inode to describe the file (governed by inode count) These are separate limits. df -h only shows data block usage. It says nothing about inodes. The Classic Scenario Here's a real example I saw last month on a production server hosting a Node.js application: $ df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 100G 85G 15G 85% / $ df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda1 6553600 6553600 0 100% / The disk is 85% full but has zero inodes remaining. That's the contradiction that confused the on-call engineer for an hour. What Causes Inode Exhaustion? Tiny Files Accumulate Every npm install, every webpack cache entry, every log rotation creates thousands of small files. A single node_modules directory can easily contain 50,000+ files. # Count files in node_modules find . -type f | wc -l # Often: 80000-150000 files Log Rotation Gone Wrong Misconfigured logrotate creates compressed copies without deleting originals. Or a backup script writes hourly snapshots to /tmp without cleanup: # Culprit pattern: backup creates a file per minute ls /backup/ | wc -l # Result: 43,200 small files Ephemeral Session Files PHP sessions, temporary upload files, or cache directories that grow without bounds. Each request creates a new file. Nothing cleans them up. # Check common inode traps ls /var/lib/php/sessions/ | wc -l ls /tmp | wc -l Diagnosing the Culprit Step 1: Confirm Inode Exhaustion df -i If any partition shows 100% in Use%, you've found the problem. Step 2: Find the Largest Directories # Top-level breakdown (slow on large filesystems) sudo du -x --max-depth=1 / 2>/dev/null | sort -rh | head -20 Focus on the directory with the most files, not necessarily the most bytes. Step 3: Identify the Pattern Once you find the suspect directory, check the file characteristics: # Check if they're tiny files (the inode killer) find /path/to/suspect -type f -size -1k | wc -l # If most files are under 1KB, that's your inode bomb The Fix: Clean Up and Prevent Immediate Relief Delete the smallest, most numerous files first. Each deletion frees one inode. Prioritize: # Clear npm cache (frees thousands of tiny files) npm cache clean --force # Rotate and compress old logs sudo journalctl --vacuum-time=3d # Remove temporary files older than 7 days find /tmp -type f -mtime +7 -delete Long-Term Prevention Configure logrotate properly so old logs compress and rotate instead of piling up: /var/log/myapp/*.log { daily rotate 7 compress delaycompress missingok notifempty create 0640 www-data adm postrotate systemctl reload myapp endscript } Then add a cron job that warns you before inodes hit critical: 0 6 * * * df -i | awk 'NR==1{next} $5 ~ /[9][0-9]%/{print "INODE ALERT: " $0}' | mail -s "Inode Usage Critical" admin@example.com When Cleanup Isn't Enough Sometimes the filesystem was created with a small inode count. Check your filesystem creation parameters: tune2fs -l /dev/sda1 | grep -i inode # Inode count: 6553600 If your inode count is too low for your workload, you have three options: Resize the filesystem (risky, requires unmount or online resize depending on filesystem type) Move data to a larger partition with more inodes Recreate the filesystem with higher inode density (last resort) For ext4, the inode ratio is controlled at format time: # Default: 1 inode per 16KB of space mke2fs -T default /dev/sda1 # For file-heavy workloads, increase inode density mke2fs -i 8192 /dev/sda1 # 1 inode per 8KB The Key Takeaway df -h tells you about disk space. df -i tells you about inodes. Run both. When you see "No space left on device" on a partition that clearly has free space, check inodes before you panic about disk capacity. The fix is rarely more storage — it's usually deleting thousands of tiny files you didn't know were piling up. More ops notes and server reliability guides: jiunotes · ops notes