← Back to the library
OperationsCore · 7 min

Writes fail with "No space left on device"

“Uploads, builds, or the database start failing with ENOSPC. Sometimes df shows free space, or deleting a huge log file frees nothing.”

ERROR CODES YOU MAY SEE

A code is a clue. Use its meaning and surrounding evidence to narrow the cause.

ENOSPCLinux errno · standard

No space left on device.

Run both df -h and df -i: running out of inodes returns the same error with free bytes left. If du totals less than df reports, look for deleted files that are still open (lsof +L1).

Official reference for ENOSPC (opens in new tab)Code definition reviewed

THE PRINCIPLE

Space is freed only when the last open handle closes. Every writer needs a cap, and every cap needs an alert before the disk reaches 100%.

FIRST MOVES

  1. Run df -h and df -i. Full inodes with free bytes means millions of small files (sessions, cache entries, mail queues).
  2. Find the growth with du -xh --max-depth=1 / | sort -h, staying on one file system. Check /var/lib/docker, /var/log, and the journal.
  3. If du totals much less than df, list deleted-but-open files with lsof +L1, then restart or signal the holder. Don't rm a live log: truncate it with : > file.
  4. Docker: docker system df, then docker builder prune and docker image prune -a. The -a removes every image no container uses, so later starts pull them again. Volumes are left alone unless you ask for them, so check before passing --volumes.
  5. Cap the source: json-file max-size/max-file (unlimited by default), logrotate with maxsize, and journald SystemMaxUse.

TOOLS: THEN → NOW

Deleting files by hand once the disk is fullSize caps per writer plus a disk/inode alert at 80-85%
logrotate copytruncate for every logReopen on signal (postrotate) where the app supports it; copytruncate can drop lines written during the copy

PATTERN SNAPSHOT

df -h /; df -i /                          # bytes vs inodes
du -xh --max-depth=1 / 2>/dev/null | sort -h | tail
lsof -nP +L1                              # deleted but still open
docker system df
journalctl --disk-usage

# /etc/docker/daemon.json: cap logs (unlimited by default; new containers only)
# { "log-driver": "json-file",
#   "log-opts": { "max-size": "50m", "max-file": "3" } }

CLOSE THE AI. EXPLAIN THIS.

You rm a 40 GB log file and df still shows the disk as full. Why, and how do you get the space back without a reboot?

SOURCES

Guide reviewed