Linux Disk Full: How to Find the Cause and Free Space Safely

When Linux reports No space left on device, immediately deleting a few large files may not solve the real problem. The filesystem may be out of data blocks or inodes, logs may be growing rapidly, Docker may retain old layers, or a process may still hold space from a deleted file. A safe response identifies the filesystem, measures the cause, recovers space deliberately, and prevents recurrence.

1. Identify the full filesystem

Begin with both capacity and inode views:

df -hT
df -ih

df -hT shows filesystem type, total, used, available, and mount point. df -ih checks inodes. At 100% IUse%, the system cannot create files even when gigabytes remain; millions of small files are a common cause.

Identify the mount that contains the failing path:

df -hT /var/lib/mysql
findmnt -T /var/lib/mysql

Always recover space on the correct filesystem. Removing 10 GB from /home does nothing when a separate /var mount is full.

2. Capture evidence before changing anything

date
df -hT
df -ih
lsblk -f
findmnt
sudo dmesg -T | tail -n 50

If the kernel reports I/O errors, a read-only filesystem, or unhealthy storage, prioritize data preservation and hardware investigation. File cleanup cannot repair a failing device.

3. Find the directory consuming space

Use du while staying on the same filesystem:

sudo du -xhd1 / | sort -h
sudo du -xhd1 /var | sort -h
sudo du -xhd1 /var/lib | sort -h

Walk down one level at a time until an unusual branch appears. -x stays within one filesystem and -d1 limits output to one level. On a busy host, schedule broad scans carefully because traversing many inodes can create significant I/O.

Locate recent large files:

sudo find /var -xdev -type f -size +500M \
  -printf '%s %TY-%Tm-%Td %TH:%TM %p\n' \
  | sort -n | tail -n 30

Do not delete a file merely because it is large. Identify its owner, active process, retention policy, and recovery path.

4. Why df and du disagree

df reads whole-filesystem allocation while du totals files visible in directory trees. They may differ because of reserved blocks, snapshots, hard links, hidden mounts, or unlinked files still open by a process.

A common case is a large log that was removed while a web server or database retained its file descriptor:

sudo lsof +L1
sudo lsof +L1 /var

Look for zero NLINK values or names containing (deleted). The normal remedy is to ask the application to reopen logs or restart the correct service after assessing impact:

sudo systemctl reload nginx
sudo systemctl restart my-application.service

Do not blindly truncate /proc/PID/fd/N for a database or unknown process. It can corrupt data or destroy incident evidence.

5. Inspect systemd journal and application logs

journalctl --disk-usage
sudo du -sh /var/log/* 2>/dev/null | sort -h

Archived journal files can be vacuumed by size or age:

sudo journalctl --rotate
sudo journalctl --vacuum-size=500M
# Or:
sudo journalctl --vacuum-time=14days

--vacuum-size directly removes only archived journals, so rotating first makes active data eligible for archival cleanup. Then set durable limits such as SystemMaxUse and SystemKeepFree in /etc/systemd/journald.conf.

For files under /var/log, fix logrotate or application retention. Removing an open log with rm can create the exact df/du discrepancy described above.

6. Inspect Docker and container storage

docker system df
docker system df -v
sudo du -sh /var/lib/docker/* 2>/dev/null | sort -h

Docker may retain images, layers, build cache, volumes, and JSON logs. Review what running containers use before pruning. docker system prune, especially docker volume prune, can remove data that is detached now but still needed for recovery or a future deployment.

Set container log limits with the local logging driver or rotation options for json-file. Validate daemon JSON and plan the restart; new settings may not automatically affect existing containers.

7. Handle inode exhaustion

If df -i is full while byte capacity remains, find directories with very high file counts:

sudo find /var -xdev -printf '%h\n' \
  | sort | uniq -c | sort -n | tail -n 30

Common sources include sessions, cache, mail queues, thumbnails, temporary files, and jobs without cleanup. Avoid a shell wildcard over millions of names. Use find with an age condition and inspect a sample first:

find /path/to/cache -xdev -type f -mtime +7 -print | head
find /path/to/cache -xdev -type f -mtime +7 -delete

Run deletion only after confirming that the sample is correct and that the data is reproducible cache.

8. Package caches, old kernels, and temporary files

On Debian and Ubuntu, inspect and clean package caches through the package manager:

sudo du -sh /var/cache/apt/archives
sudo apt clean
sudo apt autoremove --purge

Review the autoremove proposal before approving it, particularly on systems with manually installed kernels, drivers, or packages. Clean /tmp through systemd-tmpfiles policy or age conditions instead of emptying it while applications run.

9. Emergency recovery in lower-risk order

1. Pause the known workload that is rapidly generating data.
2. Rotate and vacuum archived journals.
3. Remove package cache and reproducible artifacts.
4. Handle logs through reload or logrotate.
5. Remove verified-unused container images and build cache.
6. Restart services holding deleted files according to an impact plan.
7. Expand the filesystem or volume when valid data growth is the cause.

Recover a meaningful safety margin before resuming databases and services. A few free megabytes may only allow another burst of logs before the host fills again.

10. Expand capacity when growth is legitimate

If the workload needs all current data, cleanup only delays failure. Determine whether storage is a partition, LVM logical volume, cloud disk, or standalone filesystem. A typical workflow expands the device or logical volume and then grows the filesystem with the appropriate tool, such as resize2fs for ext4 or xfs_growfs for XFS.

Create a tested backup or snapshot, verify the exact device, and read platform documentation before changing partitions or volumes. Never copy a storage expansion command from another host and run it unchanged.

11. Prevent recurrence

• Alert on both percentage and absolute free space.
• Monitor inodes, growth rate, predicted time to threshold, and read-only state.
• Limit journal, application, and container logs.
• Define retention for backups, artifacts, uploads, caches, and snapshots.
• Separate fast-growing data from the root filesystem when appropriate.
• Test alerts and cleanup procedures in staging.
• Record the root cause, not just the number of bytes removed.

Incident checklist

1. Use df -hT and df -ih to distinguish blocks from inodes.
2. Confirm the mount with findmnt -T.
3. Walk directories with du -x without crossing filesystems.
4. If df greatly exceeds du, inspect lsof +L1.
5. Measure journals, logs, and containers before cleanup.
6. Never guess-delete databases, volumes, snapshots, or open logs.
7. Measure after every change and verify services.
8. Add limits, retention, and alerts to prevent recurrence.

Conclusion

A full Linux disk is a diagnostic problem, not a file-deletion contest. df identifies filesystem pressure, du locates visible data, lsof reveals open deleted files, and journal or container tools explain common growth sources. Once the cause is clear, clean up deliberately or expand storage, then add monitoring so the next warning arrives before the outage.

References

• GNU Coreutils: df
• GNU Coreutils: du
• Linux manual: lsof
• systemd: journalctl