Sanoid and Syncoid: ZFS Snapshots on Autopilot
A policy engine for snapshots, and a replication tool that just works
Contents
ZFS snapshots are so cheap that the sensible thing to do is take one every fifteen minutes and never think about it. They cost nothing at creation — a snapshot is a reference-count bump and a pointer to the current uberblock — and they only start consuming space as the live filesystem diverges from them. Taking one is instantaneous on a dataset of any size. The temptation to take a lot of them is entirely correct.
The trouble starts a fortnight later, when your pool has 1,400 snapshots and zfs list -t snapshot scrolls for a minute, and you can’t remember which of the two cron one-liners you wrote is the one that deletes things, and you’re not confident either of them handles the month boundary. Every ZFS user writes that cron job. Every ZFS user then writes the retention half badly, because retention is where all the actual difficulty lives.
Sanoid is that cron job, written properly by somebody else. It’s a policy engine: you declare how many hourlies, dailies and monthlies you want per dataset, and it takes and prunes to match. Its companion syncoid does the other half — replicating datasets to another pool or another machine over SSH, incrementally, resuming when interrupted, without you ever composing a zfs send | ssh | zfs recv pipeline by hand. They’re a pair of Perl scripts from Jim Salter, they’ve been stable for years, and they are the least glamorous and most reliably useful thing in my storage stack.
Why a policy engine rather than a cron one-liner
The naive version looks like this, and I ran a version of it for about a year:
| |
It works until it doesn’t, and the ways it doesn’t are instructive. head -n -24 assumes lexical sort order matches chronological order, which it does until you change the date format. xargs zfs destroy with no guard will happily destroy a snapshot that a replication job is mid-way through sending. There’s no per-dataset policy, so your media library and your database get identical treatment despite having nothing in common. There’s no concept of a snapshot being needed by something else. And the whole thing fails silently, because a grep that matches nothing exits non-zero into a void.
Sanoid replaces all of that with declarations. It knows about hourly, daily, weekly, monthly and yearly buckets independently, so “24 hourlies, 30 dailies, 3 monthlies” means exactly what you’d hope. It prunes by the snapshot’s actual creation property rather than by parsing its name. It refuses to destroy a snapshot that’s marked as held. And it’s idempotent — running it every minute is harmless, which is what lets you drive it from a timer without any locking logic of your own.
The deeper reason to use it is that retention policy is a thing you should be able to read. Six months from now, the question “how far back can I go on this dataset?” should have an answer you can look up in a file instead of reconstructing from shell arithmetic. If you’re still building your mental model of what snapshots are and how they interact with scrubs and resilvers, ZFS for mortals is the right prerequisite for this piece.
The config file
Sanoid’s entire interface is /etc/sanoid/sanoid.conf. It uses templates, which is what stops it becoming repetitive:
| |
Read template_receive carefully, because it’s the one that catches people. autosnap = no with autoprune = yes means “don’t create snapshots on this dataset — something else does — but do enforce retention on what arrives”. That’s precisely what you want on a replication target: the source decides what exists, the target decides how long it keeps it. Having a backup target with a longer retention than the source is the entire point, and it’s the difference between a replica and a backup.
The other subtlety is recursive = yes, which snapshots child datasets too. ZFS has two flavours of recursive snapshot: separate snapshots per child, or one atomic snapshot across all of them (zfs snapshot -r). Sanoid’s recursive = yes does per-child; zfs_recursion = yes does the atomic version. For datasets that need to be consistent with each other — a database’s data and WAL on separate datasets, say — you want the atomic one, and getting this wrong produces a set of snapshots that individually restore fine and collectively restore to an inconsistent moment.
Then a timer, and this is the whole of the operational surface:
| |
The packaged unit runs sanoid --cron, which is take-then-prune in one pass. Run sanoid --cron --verbose --readonly first and it will tell you exactly what it would do without doing any of it. That dry-run flag has saved me from at least one policy that would have pruned rather more enthusiastically than I intended. If systemd timers are unfamiliar territory, writing your first service unit covers the shape of it.
Syncoid, and the part where it becomes useful
Snapshots on the same pool protect you against mistakes. They protect you against nothing else — the pool is a single point of failure, and every snapshot on it dies with it. The second half of the setup is getting those snapshots onto different hardware:
| |
That’s the whole command. Syncoid works out the most recent snapshot the two ends have in common, sends the increment between there and now, and if there’s no common snapshot it does a full send. It resumes interrupted transfers using ZFS’s own resume tokens. It’ll pipe through mbuffer and pv if they’re installed, and use lzop compression on the wire by default, which on a gigabit LAN with compressible data is a real speedup.
--no-sync-snap tells it to replicate the snapshots sanoid already made rather than creating its own throwaway one each run. That’s what you want when sanoid owns the policy: the target ends up holding the same named snapshots as the source, and template_receive expires them on its own schedule. Without the flag you get a litter of syncoid_* snapshots and two systems with opinions about retention.
Pull over push, if you have the choice. A push setup means the source machine holds credentials that can write to the backup target — so anything that compromises the source can reach the backups. A pull setup means the backup host holds a read-only-ish key to the source, and a compromised source can’t touch what’s already been pulled. It’s the same reasoning as everywhere else in security: the blast radius of the more-exposed machine should be as small as you can make it. Restricting that key with command= in authorized_keys closes it further, along the lines of hardening SSH beyond the basics.
Then wrap it in a timer with a dead-man’s-switch ping, exactly as you would any other backup job, because a replication that silently stopped three weeks ago is the most common way this setup fails.
Consistency, and the thing snapshots cannot see
A ZFS snapshot is atomic and instantaneous at the block layer, and that is a much weaker promise than people hear. It means the snapshot captures a coherent moment in the pool’s state. It says nothing about whether the applications writing to that pool had finished what they were doing.
For most data this is irrelevant. A photo library, a media directory, a pile of documents — a snapshot mid-write catches at worst one half-copied file, and you’d notice. For anything with its own on-disk state machine, it matters enormously. Snapshot a running Postgres data directory and what you have captured is equivalent to the state after a power cut. Postgres is built for that and will replay its WAL and come up clean, which is genuinely reassuring right up until you learn the caveat: it’s only true if the entire data directory, including WAL, was in the same atomic snapshot. Split across two datasets snapshotted a second apart, you have captured a moment that never existed, and the recovery is undefined.
This is why the zfs_recursion = yes distinction earlier is more than a footnote, and why the pre_snapshot_script hook exists. The hook runs before the snapshot and the post hook runs after, so you can quiesce whatever needs quiescing — pause VMs, flush and lock a database, whatever the application’s own documentation asks for. The script must be fast, because everything is stalled while it runs, and it must be reliable, because sanoid takes the snapshot regardless of whether your pre-hook succeeded unless you tell it otherwise with no_inconsistent_snapshot.
My own line on this: use application-native dumps for anything with a WAL, snapshot everything else, and stop trying to make one mechanism do both jobs. A nightly pg_dump into a dataset that then gets snapshotted costs a few minutes of CPU and removes an entire category of “will this restore?” uncertainty. Snapshots are a time machine for files; databases want a backup that understands them.
What actually goes wrong
“cannot receive incremental stream: destination has been modified since most recent snapshot”. Something wrote to the target dataset. Usually you mounted it and something touched it, or an application is running against it. Set readonly=on on the receiving dataset and this stops happening forever. Recovery is zfs rollback on the target to the last received snapshot, then resume.
Syncoid does a full send every time. The two ends have no snapshot in common, which means retention on one side expired the last common snapshot before the next run. It shows up when the source’s hourly retention is shorter than the gap between replication runs. Either replicate more often or keep more hourlies; the numbers have to overlap.
Sanoid isn’t pruning. Check autoprune = yes is actually on the template in use, and check nothing is holding the snapshots — zfs holds will tell you. Syncoid places holds during a transfer, and an interrupted transfer can leave one behind. zfs release -r clears it.
The pool fills up anyway. Snapshots pin the blocks that the live filesystem has since freed. A dataset that rewrites a lot of data — a VM image, a busy database — can have a usedbysnapshots figure vastly larger than the dataset itself. zfs list -o space breaks it down honestly, and the answer is usually a shorter retention on that specific dataset rather than a bigger pool.
Deleting snapshots doesn’t free space. Because of how ZFS accounts for blocks shared between snapshots, destroying one snapshot may free almost nothing — the blocks are still referenced by its neighbours. zfs destroy -nv tank/data@snap1%snap9 shows what a range destroy would actually reclaim before you commit to it.
Everything is slow after enabling frequent snapshots. frequently = 4 on a dataset with heavy random writes means four snapshots an hour each pinning a different generation of blocks, and your pool’s free space becomes fragmented. Most homelab datasets do not need sub-hourly granularity, and the ones that do are usually better served by replication to a second pool than by more snapshots on the first.
The honest verdict
Sanoid and syncoid are worth installing on the first day you have a ZFS pool with anything you care about on it. The setup is one config file and two timers, it’s an hour of work, and afterwards a category of anxiety leaves your life permanently. There’s no daemon, no database, no web UI, no dependencies beyond Perl and ZFS itself. When it breaks it breaks in a way you can read.
The limitation is the one I’ll keep repeating: this is not a backup system, and treating it as one is the mistake it makes easy. Replicated snapshots on a second pool in the same flat are copy two on a different medium, which is genuinely useful and covers a dead disk, a dead pool and a fat-fingered rm. They do not cover a fire, a burglary, or a ransomware event that reaches both machines through the credentials that connect them. The argument in snapshots are not backups applies with full force here, and syncoid’s convenience makes it easier to believe you’ve solved a problem you’ve only moved.
Use it for the granularity — a snapshot every fifteen minutes is a magnificent thing to have when you delete the wrong directory — and put something offsite, ideally something that doesn’t speak ZFS, underneath it. If your pool already has its own box and you’re running TrueNAS, you have a version of this in the UI already; sanoid is what you want when the pool lives on a general-purpose Linux machine and you’d rather the policy lived in a file you can put in git.
Related Content
