Snapshots Are Not Backups: A Cautionary Tale
How a folder full of pristine ZFS snapshots taught me nothing about recovery until the pool that held them died
Contents
For about two years I told anyone who would listen that I had backups sorted. I ran ZFS, I took hourly snapshots, I kept a fortnight of them, and I had personally watched myself roll back a botched database migration in under a minute by cloning a snapshot from an hour earlier. I felt smug. I had automated point-in-time recovery, it worked, and I stopped thinking about the problem.
Then the pool died, and it took every last snapshot with it, and I learned the difference between a snapshot and a backup the way most people learn it: at 11pm, with a cold feeling in my stomach, staring at a zpool status output I did not want to be true. This is that story, and the fix that came out of it, because the mistake I made is the single most common one I see in homelabs that think they are safe.
What a snapshot actually is
A snapshot is a point-in-time reference inside a filesystem. On ZFS or Btrfs it costs almost nothing to take, because the filesystem is copy-on-write: taking a snapshot just tells the pool “do not free the blocks that make up the current state.” As you change files, the old blocks are retained instead of recycled, and the snapshot lets you read the filesystem exactly as it was at that instant. It is a genuinely brilliant feature and I would not run a homelab without it.
Here is the crucial property, the one I had internalised as a convenience and never thought through as a risk: those retained blocks live on the same disks, in the same pool, under the same controller, in the same machine, as the live data. A snapshot is a second view of the same physical bytes. It shares fate with everything around it. Whatever kills the pool kills the snapshot at the same instant, because they were always the same storage.
Snapshots defend beautifully against a specific class of disaster: you deleted the wrong directory, an upgrade corrupted a config, a database migration went sideways, ransomware encrypted your files but did not have the privileges to destroy pool history. In every one of those cases the underlying disks are healthy and the pool is intact, so the old state is still sitting there to roll back to. That is a real and valuable protection, and it is why I recommend snapshots to everyone.
They defend against nothing that destroys the storage itself. Disk failure beyond your redundancy. A controller that writes garbage to every drive at once. A power event that takes out the whole machine. Theft. Fire. A zpool destroy typed against the wrong pool at 11pm. In all of those, the live data and every snapshot vanish together, because they were never independent copies.
The night it went wrong
My pool was a mirror of two disks — sensible redundancy, or so I thought. What I had not accounted for was that both drives came from the same batch, bought on the same day, and had therefore lived identical lives with near-identical wear. When the first one started throwing read errors, the pool went DEGRADED and I did the right thing: ordered a replacement. What I did not expect was the second, identically-worn drive to fail during the resilver, while it was under exactly the heavy sustained read load that resilvering imposes.
| |
That last line is the whole lesson in one sentence. My fortnight of pristine hourly snapshots was inside the pool that had just faulted. Every rollback point I had been so proud of was as inaccessible as the live data, for the same reason, at the same moment. I had built excellent protection against the mistakes I make with my hands and no protection at all against the disks failing under my feet. The two-disks-same-batch story is its own lesson, which I got into in rethinking RAID when mirrors beat parity in a homelab — redundancy correlated by manufacturing date is barely redundancy at all.
What saved me, partially, was that some of that data also lived on a second machine for unrelated reasons. What I lost for good was everything that had only ever existed on tank, protected by snapshots I had genuinely believed were backups. The gap between “I have snapshots” and “I have backups” cost me a chunk of data and a lot of sleep.
The rule I should have followed
A backup is an independent copy of the data on separate media, ideally in a separate machine, ideally in a separate building. The word doing the heavy lifting is independent: the copy has to be able to survive the events that destroy the original. A snapshot fails that test on every axis except one, because it lives on the same media. This is the whole reason the 3-2-1 rule exists and insists on multiple media types and an off-site copy. Snapshots are a wonderful feature of one of your copies. They never, by themselves, constitute one of the three copies.
The good news is that copy-on-write filesystems make it genuinely easy to promote snapshots into real backups, because the same snapshot machinery that gives you cheap rollback also gives you cheap, efficient, block-level replication to somewhere else.
Turning snapshots into backups with send/receive
ZFS send serialises a snapshot into a stream you can pipe anywhere, and receive reconstructs it into a pool on another machine. Send an incremental between two snapshots and you ship only the blocks that changed, which is what makes nightly off-machine replication cheap. The full mechanics deserve their own write-up, and I gave them one in ZFS send/receive: time travel for a home filesystem, but the shape of it is short enough to show here.
First, take a named snapshot on the source:
| |
Send it to a second machine over SSH, into a separate pool called backup on a different box entirely:
| |
Now the same dataset exists as a real filesystem on separate disks, in a separate machine, that can survive tank faulting. Tools like syncoid (from the sanoid project) wrap this pattern, tracking the last common snapshot automatically and handling the incremental bookkeeping so you do not have to name snapshots by hand:
| |
Run that on a systemd timer and your snapshots have become backups, because there is now a copy whose fate is not tied to the source pool. Put the receiving machine off-site — a Pi at a relative’s house will happily receive ZFS streams — and you have the off-site leg too.
Do not stop at “the replication ran”
The trap I fell into with snapshots has a second-order version with replication: believing the copy is good because the job reported success. zfs send | receive can complete and still leave you with a dataset you have never actually mounted and read. The only proof that a backup works is a restore, so mount the received dataset on the backup host periodically and check that real files open:
| |
I now run a monthly restore drill against the replicated copy, because the whole point of this article is that I once trusted a recovery mechanism I had never truly tested under the conditions that mattered. Rehearsing the recovery you dread is a discipline of its own, and it is the part almost nobody does until they have already been burned once. The drill is cheap: clone the received dataset on the backup host, mount the clone read-write so nothing touches the real replica, open a handful of known files, and diff their checksums against a manifest you generated on the source. Fifteen minutes a month buys you certainty that the incremental chain is intact and the data is genuinely readable, which is worth far more than the warm feeling of a green “job succeeded” line in a log.
Troubleshooting the send/receive path
cannot receive incremental stream: destination has been modified. The receiving dataset changed since the last snapshot, usually because something wrote to it or it was mounted read-write and touched. Receive with -F to roll the destination back to the last received snapshot, or set the backup dataset readonly=on so nothing can modify it between syncs.
could not find any snapshots to destroy; check snapshot names during pruning. Your retention on source and destination has drifted, and the last common snapshot you need for the next incremental has been expired on one side. Keep at least one common snapshot on both ends; syncoid manages this, but hand-rolled scripts often prune too aggressively and break the incremental chain, forcing a slow full resend.
The stream is slow over the network. zfs send is CPU-bound on compression and the pipe is limited by your uplink. For the first full send, replicate locally over the LAN, then physically relocate the disk. Enable send -c to send already-compressed blocks without recompressing when your source dataset uses compression.
Snapshots piling up and filling the pool. Snapshots retain old blocks, so heavy churn plus long retention eats space fast. zfs list -t snapshot -o name,used shows which snapshots are pinning the most data. Set a sane retention policy with sanoid rather than keeping snapshots forever.
Verdict: keep the snapshots, add the backups
Nothing here is an argument against snapshots. I still take them hourly, I still roll back botched changes in seconds, and I would not give that up. The correction is narrow and it is the whole point: a snapshot protects the current state of a filesystem against your mistakes, and it protects nothing against the disks dying. Treat snapshots as the fast, local undo button they are, and build genuine backups — independent copies on separate media, replicated off the machine, verified by restore — as a separate discipline sitting underneath them.
Who is this for? Anyone running ZFS, Btrfs, LVM thin snapshots, or a NAS that advertises “snapshot protection” on the box and lets you feel covered. If your recovery story begins and ends with snapshots that live on the same pool as your data, you are one bad resilver away from the 11pm I had. Take the twenty minutes to set up syncoid to a second machine tonight, and then, next weekend, actually restore from it. The smugness I felt for two years was real. It was also completely unearned, and the pool did not care how confident I was.
Related Content
