Restic and Rclone: A 3-2-1 Backup Pipeline That Actually Restores
Two tools, three copies, and the rehearsal nobody does until it is too late
Contents
Everybody who self-hosts has a backup story, and the honest ones are all the same story: the backups were running for two years, and the first time anyone tried to restore from them was the day they were needed, and that was the day everyone found out what “running” had actually meant. Mine involved a repository that had been dutifully backing up an empty directory since a path change eight months earlier. The cron job exited zero every night. Nothing was wrong, except everything.
So this is the pipeline I settled on, and more importantly the drill that goes with it. Restic makes the snapshots. Rclone moves them somewhere that isn’t my flat. Systemd runs the thing and shouts when it doesn’t. And once a month, a script restores a random file from a random snapshot and compares it byte for byte with the original, because a backup you have never restored from is a belief rather than a fact.
What 3-2-1 is actually asking of you
Three copies of the data, on two different media, one of them offsite. It’s a rule from the photography world and it has survived because each clause defends against a genuinely different failure, and it’s worth being precise about which is which, because most homelab setups quietly satisfy one clause three times and think they’ve done all three.
Three copies defends against the ordinary case: a drive dies, a file is deleted, a database eats itself. Two media defends against the correlated case — a batch of drives from the same production run failing in the same month, or a controller that scribbles on everything attached to it. One offsite defends against the total case: fire, theft, flood, or a burst pipe in the flat above. That last one is the one people skip, and it’s the only one that covers the scenario where you lose everything at once.
The clause nobody writes down is the one that actually matters. Ransomware and your own fingers both count as failures, and both propagate through anything that syncs. A mirror is not a backup. A ZFS pool is not a backup. rclone sync of your data directory to cloud storage is not a backup — it’s a very efficient way of replicating a deletion. I’ve made this argument at more length in snapshots are not backups, and it’s the whole reason the pipeline below has restic in the middle rather than just an rclone job.
RAID is in the same category and gets confused for backup more often than anything else. A mirrored pool survives a dead disk beautifully and offers precisely nothing against rm -rf, which is the actual thing that happens to people. Mirrors versus parity is a question about availability; this is a question about recovery, and they are different departments.
Why restic, and why rclone underneath it
Restic gets three things right that matter for this job. It’s content-addressed and deduplicating, so the fiftieth snapshot of a 200GB dataset costs whatever actually changed. It encrypts client-side with no option to turn it off, which means the offsite copy can live somewhere you don’t trust without any further thought. And a restic repository is a directory of opaque blobs — which means anything that can move a directory can move a repository.
That last property is what makes the rclone pairing work. Restic has native backends for S3, B2, SFTP and a few others, and they’re fine. But rclone speaks about seventy storage providers, and pointing restic at a local repository and then rcloning that directory offsite gives you a clean separation: restic owns correctness, rclone owns transport. When your offsite provider does something annoying — rate limits, a pricing change, an outage — you change the rclone remote and restic never notices. The backup logic and the transport logic fail independently, which is the property you want at 3am.
The alternative shape is restic writing straight to a remote via rclone: as its backend, which is one fewer moving part and one fewer copy. I run the two-stage version because the local repository is copy two, and restoring from it takes seconds rather than a download. If you’ve already read Borg vs restic, the same reasoning applies: restic’s willingness to treat a repository as just bytes is exactly what buys the flexibility here.
The pipeline
Repository on a separate pool from the data. Password in a file root can read, and — this is not optional — also somewhere you can reach when the machine is ash. A restic password stored only on the machine being backed up is a very elaborate way of having no backups at all. Mine lives in my password manager and on a piece of paper in a different building. If you’re keeping it in git alongside your configs, sops and age is the way to do that without regret.
| |
The backup script. It’s short, and every line in it is there because something went wrong once:
| |
The order matters. Database dumps before the snapshot, because restic copying a live Postgres data directory gives you a snapshot of a torn write — it restores, it starts, and it is subtly wrong, which is worse than failing. Dumps, WAL and PITR covers what “properly” looks like for anything you care about. forget --prune before the rclone, so you’re not shipping blobs you’re about to delete.
Then systemd, because cron will not tell you when it fails:
| |
| |
Persistent=true catches up a run missed while the machine was off, which cron cannot do. IOSchedulingClass=idle stops the backup from making everything else feel broken at 2:30am. The curl calls at either end are a dead-man’s switch — something has to notice the absence of a backup, and neither cron nor systemd will do that for you. Self-hosted dead man’s switches is the pattern; the point is that silence must be an alarm.
Choosing the offsite end
The rclone remote is the decision with the longest tail, because it’s the one you’ll live with for years and the one that costs money every month. Three shapes are worth considering, and the right answer depends almost entirely on how much data you have.
Object storage priced per gigabyte with cheap egress is the default. The thing to check is retrieval cost rather than storage cost, because storage is quoted loudly and egress is quoted quietly, and the day you need this pipeline is the day you download all of it at once. A provider charging nothing to store and a fortune to retrieve has sold you a lottery ticket. Run the sum for a full restore of your actual dataset before you commit, and if that number makes you wince, you have not got a backup — you have got an expensive hope.
A machine at somebody else’s house is the option I keep coming back to for bulk data, and I’ve written up the shape of it in an offsite Pi at your sister’s. The economics are unbeatable and the operational cost is real: it’s a computer you can’t easily touch, on a network you don’t control, behind an ISP router somebody might unplug. If you go this way, monitoring it matters more than monitoring anything in your own rack, because a dead offsite node fails silently and stays dead until you need it.
Rotating physical disks is the unfashionable answer and remains the best one for large archives. An encrypted external drive that lives at work, swapped monthly, is genuinely offsite, genuinely air-gapped, and immune to every network-borne failure including the ransomware case. It fails on recency — a month-old copy is a month-old copy — which is why it works best as copy three underneath something continuous rather than instead of it.
Whichever you pick, encrypt before it leaves. Restic already has, which is the quiet reason this pairing works: rclone is moving blobs it cannot read, so the trust question about the far end mostly evaporates. The remote holds ciphertext and a directory structure that leaks file sizes and timings and nothing else.
The part everyone skips
Here’s the drill. It runs on the first of the month, it takes four minutes, and it is the only part of this whole article that would have saved me from my own empty-directory fiasco:
| |
restic check alone verifies the repository’s structure — that every blob referenced by an index exists. --read-data-subset=5% makes it re-read and re-hash a rotating slice of the actual data, which is what catches bit rot and a storage backend quietly returning garbage. Over a year at 5% a month you’ve verified about half the repository, at a cost you’ll never notice.
The cmp at the end is the bit that matters, and it’s the bit that would have caught my dead pipeline. It asks the filesystem whether the bytes came back. Once a year, do the bigger version too: restore the whole thing to a spare disk and boot it, which is the argument in testing your backups.
What actually goes wrong
“repository is already locked exclusively”. A previous run died mid-prune. restic unlock clears stale locks. If it happens regularly, two jobs are overlapping — systemd won’t start a second instance of a oneshot service, which is one more reason to use timers over cron.
The prune takes four hours. Restic’s prune rewrites pack files to reclaim space and it is genuinely slow on large repositories over a network. Keep the repository local and let rclone move the result; that’s a large part of why this pipeline has two stages.
rclone sync deleted my offsite copy. sync makes the destination match the source. If the local repository is destroyed and the job runs before you notice, it faithfully propagates that. Two defences: --backup-dir so deletions move aside rather than vanish, and object-lock or versioning on the bucket so the provider refuses deletion for N days regardless of what your script asks. If your offsite target is your own S3-compatible store on a machine at a relative’s house, enable versioning there too.
Excludes that silently swallow everything. restic backup --verbose prints the file count. If tonight’s number is dramatically smaller than last night’s, something is wrong. My empty-directory incident lived in exactly that gap: a job that succeeded, quickly, on nothing.
Restore is slow from cold storage. Glacier-class tiers price retrieval, in money and in hours. Fine as copy three; a poor choice for the copy you’d actually reach for.
Memory during prune and check. Restic holds a good deal of index in RAM, and on a repository with millions of small files it will cheerfully OOM a 2GB box. If your backup host is a Pi and your dataset is a photo library, expect to meet this. The fixes are boring and effective: exclude the caches and thumbnails nobody needs restored, or move the repository host to something with more headroom. restic --limit-upload and --cache-dir on fast local storage both help more than you’d expect.
The repository grows and prune reclaims nothing. Almost always retention that isn’t doing what you read it as. --keep-daily 7 keeps seven daily snapshots; it does not delete anything older on its own, and forget without --prune marks snapshots as forgotten while leaving every byte in place. restic snapshots after a run tells you what survived, and if the list is longer than your ladder implies, your tags and your forget flags disagree.
Backing up the backup. A tempting mistake is including /srv/backup in the source paths, which snapshots the repository into itself and grows without bound. The excludes file earns its keep here, and it’s worth reading it out loud once a year.
The honest verdict
This is the right shape for anyone whose data lives on one or two Linux boxes and who wants recovery to be boring. Two tools, both of which do one thing, both of which have been around long enough to have their sharp edges filed down. The whole thing is about eighty lines of shell, and eighty lines is a size you can read on the worst day of your year.
It’s the wrong shape if you’re backing up a fleet, or if you want a UI and a dashboard. Kopia has a better story there and I’ve said so in the piece on it. It’s also more than you need if everything you own already lives on a ZFS pool and your offsite is a second pool — send/receive is a shorter path to the same place, with the caveat that it ties both ends to ZFS forever.
The part I’d defend hardest is the drill. The pipeline is unremarkable and a dozen articles describe something similar. The monthly cmp is the only thing standing between you and finding out, at the worst possible moment, that “the backups have been running fine” was a sentence about a cron job’s exit code rather than about your data.
Related Content
