I've got a Disk Space/Free Space/Out of space error but df tells me differently!

I've got a disk space error message but df shows plenty of space! What is wrong here?

Some error messages relate to disk free space but may be due to running out of inodes. Each file and directory requires an inode entry - and ext2/ext3/ext4 filesystems sometimes run out of inodes due to a large number of files being present.

Kubelet log may have keywords like  [{"type":"OutOfDisk"}...{"type":"DiskPressure"}...]

Use df -i to view inodes:

[user@host ~]$ df -ih /
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/fedora-root 2,1M 401K 1,7M 20% /

The above example tells us about 401K files are present, but inode usage is only at 20% (safe) and could store about 5x the number of files. The disk space however may not:

[user@host ~]$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/fedora-root 32G 19G 12G 62% /

The disk space can only grow about 50% more.

How do I remedy the inode usage?

Typically you will have a process that creates a lot of files. Some instances are log files, mail files that cannot be sent, or dead-letter messages (Domino sometimes has these).

Use this command to locate the most populated folder on a filesystem, i.e. /var:

{ find /var -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n; } 2>/dev/null
...
 98 /var/lib/dnf/yumdb/s
190 /var/lib/dnf/yumdb/g
313 /var/lib/dnf/yumdb/p
559 /var/lib/dnf/yumdb/l
1023 /var/log

In this instance, the 'largest' folder is /var/log but your problematic folder will contain hundreds of thousands of files.

Once you've identified the folder, you should attempt to identify the content and on that basis purge, delete, compress or otherwise reduce the number of files. Deleting over 10K files may require it be done in steps or automating as '*' will not work with too many files on the command line.