The 3-2-1 Rule, Actually Implemented at Home
Three copies, two media, one offsite — and the tools that make it real instead of aspirational
Contents
Everyone who has spent five minutes reading about backups can recite 3-2-1: three copies of your data, on two different media, with one copy offsite. It is repeated so often that it has become wallpaper — a phrase people nod at without ever building it. I nodded at it for about three years before I actually laid out, disk by disk and cron job by cron job, which piece of my setup satisfied which leg of the rule. The exercise was uncomfortable, because it turned out my “offsite backup” was a second drive in the same server chassis, which is not offsite by any definition that survives a house fire.
This post is the concrete version: what 3-2-1 means when you translate it from slogan to hardware and software you can point to, what actually counts as a separate “medium,” and how I test that the whole chain still works rather than just trusting that it does.
Why the rule exists, in one incident
The 3-2-1 rule is a defence against correlated failure. A single copy dies to disk failure, ransomware, accidental deletion, or fire — four different failure modes, and a single copy is defenceless against all of them. Two copies on the same disk still both die when that disk dies. Two copies on two disks in the same machine still both die in a power surge, a ransomware sweep that finds every mounted volume, or a house fire. The rule forces you to break correlation at three points: enough independent copies that a lost one doesn’t take a second one with it, different media types so a single failure mode (say, a bad firmware bug in one drive model) can’t kill everything, and physical distance so a local disaster can’t reach every copy at once.
The incident that made this concrete for me was smaller than a house fire, which is exactly why it stuck. A firmware update rolled out to a batch of drives from the same manufacturer, and a subset of that batch developed a pattern of silent read errors under sustained load, producing subtly corrupted files well before anything looked obviously missing. I had two of that exact drive model in the same array, bought together, flashed together, because buying matched pairs is the sensible thing to do for a RAID rebuild. It is also precisely the correlated risk 3-2-1 exists to break: same vendor, same firmware revision, same purchase batch, same failure trigger. RAID and a second disk gave zero protection here, because every disk sharing that specific firmware defect misbehaved in lockstep. The copy that actually saved the data was the offsite one, on different hardware entirely, running a different filesystem, untouched by the firmware issue because it had never seen that firmware.
That is the distinction worth sitting with: 3-2-1 is really about auditing what your copies have in common and deliberately breaking every shared dependency you can identify. A count of three copies that all share a vendor, a power circuit, and an administrator is a weaker defence than two copies that genuinely share nothing.
Mapping the rule onto a real setup
Here is how the three legs map onto tools I actually run, rather than abstractions.
| Leg | What it means | What I run |
|---|---|---|
| Copy 1 (production) | The live data, wherever it normally lives | ZFS pool on the main server |
| Copy 2 (local backup, different medium) | A second copy, different disk or technology, same site | Nightly Restic snapshot to a separate USB-attached disk |
| Copy 3 (offsite) | A copy that survives the loss of the entire site | Encrypted Restic snapshot pushed to an S3-compatible bucket in another region |
“Different medium” is doing real work in that table. Two ZFS pools in the same chassis are still two copies, but they are the same medium in the sense that matters: same power supply, same building, same firmware family, same administrator fat-fingering a command against both at once. A genuinely separate medium means a different device class (spinning disk vs. SSD vs. cloud object storage), ideally a different vendor, and — for the offsite leg — different physical custody entirely.
Building copy 2: a proper local backup
The production copy already lives on a ZFS pool with regular scrubs, which protects against silent bit rot and single-disk failure. But a ZFS pool is still one administrative domain: a bad zfs destroy, a ransomware process with write access to the mount, or a controller failure can take the whole pool with it. Copy 2 needs to be genuinely separate — a second disk that is not part of the same pool, ideally not even attached most of the time.
| |
Keeping the disk unmounted outside the backup window is deliberate: a drive that isn’t mounted can’t be encrypted by ransomware or wiped by a stray command, which is a meaningful chunk of “two media” done properly rather than just technically.
Building copy 3: the offsite leg
The offsite copy is the leg people skip, because it is the one with recurring cost and the one that never seems urgent until the day it is the only copy left. A cheap, reliable way to do this at home scale is object storage — Backblaze B2 or an S3-compatible provider — pushed to with client-side encryption so the provider never sees plaintext:
| |
Weekly rather than nightly is a deliberate cost trade-off: the offsite leg exists to survive a total site loss, and losing up to a week of changes in that scenario is an acceptable trade for a bill that stays under a few pounds a month. If your data changes fast enough that a week of loss is unacceptable, run it nightly and budget accordingly — the rule doesn’t specify a frequency, it specifies a topology.
A cheaper alternative to a paid bucket, if you have a friend or family member willing to host a small box, is a reciprocal arrangement: Syncthing or a scheduled Restic push over Tailscale to a machine at their house, and you host their offsite copy in return. The security model is the same as any offsite target — encrypt client-side and the host never needs to be fully trusted.
What actually counts as “two media”
A question worth being pedantic about, because it is where people fool themselves: a NAS and a cloud bucket are two media. Two different brands of SSD in the same server are not — they are one medium (local disk, same site, same power) with vendor diversity, which is a nice-to-have but does not satisfy the rule. A local disk and a tape or optical archive are genuinely two media, in the classic sense the rule was written for, though tape is rare in a home context now that cloud object storage is so cheap. The test I use: if the failure mode that kills copy 1 could plausibly also kill copy 2, they are not different enough.
Testing the whole chain
A 3-2-1 setup that has never been tested end to end is only ever a belief until you prove it. I run a quarterly test where I deliberately treat the primary as gone and restore purely from copy 3, the offsite leg, onto a spare machine:
| |
This is the same instinct behind rehearsing a full bare-metal restore instead of a file-level spot check — the goal is proving the worst-case recovery path works, with the primary genuinely unavailable rather than sitting there for a convenient comparison.
Tying it together with one script and a timer
Three separate legs run by three separate cron jobs is how the offsite copy quietly stops running without anyone noticing — the local job keeps succeeding, the dashboard looks green, and nobody checks the third line down. I wrap all three legs in a single script that exits non-zero if any leg fails, so monitoring only needs to watch one job:
| |
set -euo pipefail is doing real work here: if the USB disk fails to mount, or the S3 credentials expire, the script stops immediately rather than silently skipping a step and reporting overall success. Drive this from a systemd timer as with any other backup job, and alert on the service’s failure state — journalctl -u run-321-backup.service -p err in a nightly digest email is enough to catch a dead leg within a day rather than a quarter.
Troubleshooting
The offsite bill grows every month even with pruning. Check that --prune is actually running (some workflows separate forget and prune into different schedules) and that old snapshots referenced by --keep-* policies aren’t accumulating faster than expected. Object storage also bills for API requests, not just storage — a very frequent small-file backup can rack up request charges that dwarf the storage cost itself; batching changes into less frequent, larger snapshots is often cheaper overall.
Copy 2 and copy 3 diverge and I can’t tell which is authoritative. Neither should ever be treated as authoritative — copy 1, the production data, is the source of truth, and copies 2 and 3 are both derived from it on their own schedules. If they disagree, that’s expected; the question to ask is only “does copy 1 still exist and match what I expect,” not “which backup is right.”
The local backup disk failed and I didn’t notice for weeks. This is the single most common way 3-2-1 quietly degrades to 2-1-1. Wire up SMART monitoring on every backup target, not just the primary pool, and alert on the backup job’s own exit code, not just its schedule — a job that “ran” but failed silently every night for a month is worse than no job at all, because it manufactures false confidence.
Restoring from the offsite copy is painfully slow. This is somewhat expected — egress from cloud object storage is throttled by your own bandwidth and, on some providers, by egress fees that make them stingy with throughput. This is precisely why copy 2, the fast local backup, exists: treat the offsite leg as insurance against total site loss and reach for copy 2 as your everyday restore path. If a restore from copy 3 needs to be fast, consider a provider with free or cheap egress, or keep the offsite copy smaller by excluding easily re-downloadable data (Linux ISOs, container images) that doesn’t need to survive a fire.
Verdict: is the discipline worth it?
Yes, and the honest reason is that the failure it protects against is rare enough to forget about and severe enough to be unrecoverable when it happens. None of the individual pieces here are exotic — Restic, a spare disk, and a cheap object-storage bucket are all things most homelabbers already have lying around in some form. What actually delivers the protection is refusing to let “two copies on two disks in the same box” count as done, and refusing to let the offsite leg become the thing you meant to set up next month. Draw your own table like the one above, name the actual tool against each leg, and if any cell says “I should really get to that,” that is the gap that will hurt.
The whole exercise took me an evening: naming each copy, confirming the medium was genuinely different, wiring the offsite leg to a bucket, and writing the one script that fails loudly if any part of it stops working. An evening is a small price for a setup that turns “I think my data is backed up” into a table you can point at and a log file that proves it ran last night. Do the exercise once, write it down, and revisit it whenever you add a new machine or move a service — treat 3-2-1 as a standing checklist you keep re-running against reality.
Related Content
