A backup you haven't restored is a rumour

Ask an estate whether it has backups and you’ll get a confident yes: the job runs nightly, the dashboard is green, the repository is growing. Ask when anyone last restored one and the room goes quiet.

Here’s the uncomfortable framing: an unverified backup is not evidence — it’s a rumour. Someone once said the data was safe, the claim got repeated nightly by a cron job, and everyone chose to believe it. The green tick proves a process exited zero. It says nothing about whether the bytes at rest can be turned back into a running application on the worst day of the year.

The restore drill is the acceptance test

When I consolidated the homelab’s backups onto the NAS — every cluster’s application state streaming into one repository per cluster — the job wasn’t “done” when the first snapshot landed. It was done when a scratch pod pulled thousands of files back out of the repository, the configs checked intact, and the services proved recoverable from nothing but the backup and its key. Both clusters. Before the old backup path was retired, not after.

That’s the rule worth writing down: a backup pipeline ships with its restore drill, the same way code ships with its tests. Not a runbook that describes a restore — an actual rehearsal, on the real repository, with a pass/fail outcome.

What the drill actually catches

The failures a drill surfaces are precisely the ones the nightly tick can’t see. A repository still locked by a dead process from last week’s crashed run — every subsequent “successful” backup queued behind a stale lock. A prune step that had been silently failing. And the one that matters most on the worst day: whether the key is where the recovery plan says it is. Every one of those was invisible from the green-tick view, and every one is a nothing-burger to fix on a Tuesday afternoon — and an incident report if discovered during a real recovery.

There’s a quieter design lesson the drill enforces too: back up state, not just volumes. A database dump you can restore beats a filesystem snapshot you have to forensically reassemble; the drill makes you notice the difference, because the drill is where you actually try.

One key to rule the recovery

Consolidation concentrates risk on purpose — one NAS, one repository format, one passphrase unlocking every cluster’s history. That’s a fine trade if the key management is treated with the same rigour as the data: the passphrase lives offline, not in any repository it unlocks, and the recovery plan starts from “a bare machine and the key”, assuming nothing else survived. A backup encrypted with a key stored next to it is a padlock with the key taped to the shackle.

The principle

Trust receipts, not reports. Any system whose entire purpose is a future emergency — backups, break-glass accounts, failovers, restore paths — is untested by definition on every normal day, so you have to manufacture the test: drill it, on a schedule, with a real pass/fail, and treat a failed drill as a real incident. The nightly green tick is the rumour. The drill is the receipt.

Live across the homelab: per-cluster restic repositories on the NAS over SMB, nightly CronJobs, restore drills run before each cutover — thousands of files pulled back and verified per cluster, a stale-lock failure caught and self-healing added, the passphrase held offline.