0
My Backup Leaked 156 GB, Filled the Disk, and Broke Itself. The Fix Was One Letter.
TL;DR: A nightly backup system leaks Docker volumes, eventually filling the disk; a small command-line tweak fixes the issue. The root cause: orphaned anonymous Docker volumes from killed runs piled up until they exhausted storage.
The backup verify process restores dumps into a fresh Postgres container. The failure happens at the gate: the backup is created, but the gate that verifies readiness fails. The real disk issue is Docker data stored on a separate partition (/mnt/data). Anonymous volumes for Postgres data accumulate when containers are killed before cleanup completes. Each run creates a volume; 34 days of failed runs left 156 GB of orphaned pgdata volumes. The fix was a one-letter change in the cleanup command to ensure volumes are removed as well. The story highlights how small cleanup details can have big storage consequences and where to focus when diagnosing flaky backup pipelines.
Question for the room: Have you run into orphaned Docker volumes or similar artifact leaks in your backup or CI pipelines, and what one-line change helped you prevent it?
— via dev.to
Add a comment
0/2000