Testing Your Backups: The Restore You Never Rehearsed

There is a particular flavour of dread reserved for the moment you finally need a backup and discover it does not work. The disk is dead, the data is gone, you type the restore command with slightly shaking hands, and the tool reports that the repository is corrupt, or the passphrase you saved is wrong, or the “nightly” backup silently stopped running four months ago. Everything up to that moment felt like being protected. None of it was, because a backup you have never restored is a hypothesis about the future, and the day the disk dies is the worst possible time to test the hypothesis.

I learned this the mild way — a test restore that failed on a Sunday afternoon, with the original data still safely present, rather than a real one that failed on the worst day of the year. That is the entire argument for rehearsing restores: move the discovery of a broken backup from the emergency to the drill. This article is about doing that deliberately: the quick automated check that runs forever, the occasional full-scale recovery rehearsal, verifying integrity so corruption surfaces early, and the specific things that only ever break during a restore.

Why backups fail exactly when you need them

Backups do not usually fail loudly at creation time. They fail quietly, in ways that only become visible when you try to reverse the process, and there are a handful of recurring reasons.

The backup job stopped running and nobody noticed. A cron entry that broke after an OS upgrade, a systemd timer that was never enabled, an expired cloud token — the job stops, the last-modified date on the repository freezes, and because nothing is watching, you keep believing you are covered. The repository is intact; it is simply months out of date.

The data got there but cannot come back. Silent corruption in the repository, a truncated upload, bit-rot on the backup disk, or an incomplete final chunk means the bytes are present but unreadable. Deduplicating tools are especially unforgiving here, because a single corrupt chunk can affect every snapshot that references it. The backup looks complete in a listing and fails on extraction.

The human parts are missing. You have the encrypted repository and not the passphrase, because it lived only on the machine that died. You can restore the files and not the database, because a live database file copied under load is inconsistent and needs a proper dump. You restore the data and realise you never recorded how the service was configured, so the files are back and the application will not start. These are the failures no integrity check catches, and only a real rehearsal exposes.

The cheap check that should run forever

The lowest-effort, highest-value test is an automated restore of a known file, run on a schedule, comparing what comes back against what you know it should be. It is a handful of lines and it catches the “repository is unreadable” and “backups silently stopped” failures continuously. For a Restic repository:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

#!/usr/bin/env bash
set -euo pipefail
source /etc/restic/offsite.env

canary="/etc/hostname"          # a small, stable, always-present file
workdir="$(mktemp -d)"
trap 'rm -rf "$workdir"' EXIT

# restore just the canary from the most recent snapshot
restic restore latest --target "$workdir" --include "$canary"

# it must exist and match the live copy
if diff -q "$canary" "$workdir$canary" >/dev/null; then
  echo "restore-check OK: $(date -u +%FT%TZ)"
  curl -fsS https://hc.mylab.local/ping/restore-check   # dead-man's switch
else
  echo "restore-check FAILED" >&2
  exit 1
fi

Run it weekly from a timer. The curl to a dead-man’s-switch service is the important part — it inverts the alerting so that silence is the alarm. If the check stops pinging because the repository broke or the job died, the monitoring service notices the missing ping and tells you, instead of you finding out during a real emergency. This one script converts “I assume my backups work” into “I would have been told by now if they did not”.

Pair it with a scheduled integrity check. Restic’s restic check verifies repository structure, and restic check --read-data (or a sampled --read-data-subset) actually reads and re-verifies the stored chunks, catching bit-rot that a structural check misses. Borg’s equivalent is borg check --verify-data. Run the light check often and the full data check periodically, because corruption you find on a schedule is a re-upload; corruption you find during a restore is a loss.

The full-scale rehearsal

The canary check proves the repository is readable. It does not prove you can rebuild a working system, and those are different claims. Once or twice a year, do a genuine recovery drill: pretend the main machine is gone and rebuild a service somewhere else from the backup alone.

Pick a service. Spin up a clean VM or container with nothing on it. Now, using only your backups and your documentation, restore the data and bring the service back to life. The rule that makes this valuable is to touch nothing on the original — no copying config from the live box, no “oh I remember, it needs this environment variable” from memory. If it is not in the backup or written down, it does not exist for the purposes of the drill.

You will find gaps every single time. The database needed a proper dump, because a copy of the live file restored to garbage. The service wanted a specific directory that was excluded from the backup. The reverse-proxy config that makes the thing reachable lived somewhere you never backed up. A whole discipline of practising the from-scratch rebuild — the bare-metal restore — is worth building into your routine, and every rehearsal shortens the real recovery and lengthens the list of things you now back up. Time the drill, too: knowing a full restore takes six hours changes how you plan around an outage.

Databases deserve special mention because they are the most common thing people back up wrongly. A running database’s files are inconsistent when copied, so a file-level backup of a live database can restore to a corrupt or unopenable state. The correct approach is a proper dump or the engine’s own point-in-time mechanism, which is a topic in its own right covered in database backups done right. Rehearse a database restore specifically, because “the files are backed up” and “the database restores and starts” are very different guarantees.

Rehearse the human parts, too

A restore is only partly a technical operation, and the parts that most often sink a real recovery are the human ones. Rehearse these deliberately:

The passphrase and keys. Restore from a machine that has never held the credentials, to prove they exist somewhere independent. If the only copy of your Restic passphrase or ZFS key was on the dead machine, your encrypted backup is a brick. This alone justifies the whole exercise for anyone using client-side encryption.

The runbook. Write down, as you do the drill, the exact steps to recover each service — which repository, which snapshot, which order, what config to apply. Store that runbook somewhere that survives the disaster it describes, which means not solely on the machine being recovered. A recovery you can follow from a printed page at 3 a.m. beats one you have to reconstruct from memory while stressed.

Access to the offsite copy. Confirm you can actually reach and authenticate to the offsite target from a fresh machine. Expired tokens, IP allowlists that block the recovery site, and 2FA tied to a lost phone all surface here and nowhere else.

Troubleshooting: what the rehearsal exposes

restic check passes but a restore of one file fails. The structural check does not read chunk contents. Run restic check --read-data (or a subset) to actually verify stored data; the discrepancy means a pack file is damaged. Restic’s repair subcommands can rebuild an index or prune unreadable packs so the healthy snapshots remain restorable.

The restore is far slower than expected. Restoring from cold cloud storage, or reassembling millions of small deduplicated files, can be dramatically slower than the backup was. This is why you time the drill — discovering a restore takes a day is far better learned in rehearsal. Mitigate by keeping the fast local copy (the on-LAN leg of a proper 3-2-1 setup) as the first thing you reach for, and the offsite copy as the fallback.

Restored files have the wrong ownership or permissions. Some backup tools need to run as root to restore ownership faithfully, and a restore run as an unprivileged user lands everything owned by that user. Services then refuse to start on files they cannot read. Restore with the privileges the tool needs and verify ownership on a sample before declaring success.

A ZFS receive of your replicated copy will not mount or is read-only. A replicated dataset is often deliberately readonly=on to protect the incremental chain. To use it as a live restore, clone the relevant snapshot (zfs clone) rather than mounting the replication target directly, so you do not break the very send/receive chain you rely on for the next incremental.

The database restored but will not open. Almost always because a live database file was backed up instead of a consistent dump. Re-do the backup properly (a dump or the engine’s snapshot/WAL mechanism) and rehearse that restore. A file-level copy of a busy database is the classic backup that looks fine until the day you need it.

You restored on top of the live data and made things worse. Restoring into the original location while some of the original is still present can overwrite good current data with an older backup, turning a recoverable incident into a genuine loss. Restore to a scratch directory first, verify the contents there, then move them into place deliberately. Pointing a restore straight at production during an emergency is how a bad day becomes a catastrophe.

Everything restored, but the service will not come up. The data is back and the surrounding configuration is absent — environment files, secrets, reverse-proxy rules, systemd units. This is the gap a full rehearsal exists to find. Add the missing configuration to the backup set and update the runbook, then run the drill again until a clean machine plus the backups genuinely reconstitutes the service.

Verdict: is it worth it, and for whom?

For anyone who relies on their backups — which is anyone who has them — testing earns its keep as the step that turns a backup from a hope into a guarantee. The weekly automated canary check with a dead-man’s switch costs almost nothing and continuously catches the two most common silent failures, so at a minimum, everyone should run that. It is the highest return on effort anywhere in a backup strategy.

The full recovery drill is more work, and it is worth it in proportion to how much you would lose and how complex your setup is. A single laptop with a couple of folders needs little more than the canary check and one honest “can I actually get these files back” test. A homelab running a dozen services with databases and interdependencies earns a real annual rehearsal, a written runbook, and a proven independent copy of every key. Whichever you are, do the thing almost nobody does and restore your backup before you need to. The Sunday afternoon a test restore fails is a good day; the emergency where it fails is the day you find out the protection was imaginary all along.