ZFS Replication: Off-Site Backups with send/receive

Snapshots protect you against “I deleted the wrong file” or “an update trashed my config”. They do nothing against “the building burned down” or “someone stole the NAS”, because a snapshot lives on the same pool, the same disks, the same physical location as the data it’s protecting. I learned to take that distinction seriously after a friend’s entire homelab — snapshots and all — went into a skip along with the rest of a flooded basement. The snapshots were flawless right up until the moment the hardware they lived on stopped existing.

zfs send and zfs receive are the mechanism that gets a copy of your data genuinely off-site, and they do it in a way that’s fundamentally more efficient and more correct than file-level tools like rsync, because they operate on ZFS’s own block-level understanding of what changed between two points in time rather than having to walk the filesystem and diff file contents.

The distinction between “snapshot” and “off-site copy” is worth belabouring because it’s the single most common gap I see in otherwise careful homelab backup setups. People who’d never dream of skipping backups entirely will happily run years of ZFS snapshots as their only protection, because snapshots feel like backups — they’re versioned, they’re space-efficient, zfs rollback genuinely does undo a disaster. But a snapshot’s dependence on the pool’s own hardware is total. Every snapshot ever taken shares the same power supply, the same physical building, the same exposure to theft, fire, and flood as the live data it protects. The friend whose basement flooded didn’t lose his data because his snapshot policy was wrong; he lost it because a snapshot policy was the only policy he had.

Why send/receive Beats File-Level Sync for This

Rsync compares files by path, size, and modification time (or content hash, if you tell it to), and has to walk the entire source tree to figure out what to send even when very little has actually changed. zfs send sidesteps that entirely: given two snapshots, ZFS already knows, at the block level, exactly which blocks changed between them, because that’s precisely the bookkeeping copy-on-write does anyway. An incremental send is a stream of “the blocks that differ between snapshot A and snapshot B”, full stop — no directory walk, no stat comparison, no guessing.

The practical result is that incremental replication of a dataset with millions of small files, which would make rsync grind for ages on the tree walk alone, is nearly as fast as replicating a dataset with a few huge files, because the send only cares about changed blocks regardless of how many files they belong to. It also means the receiving side gets an exact, bit-for-bit replica, including anything rsync tends to be finicky about — ACLs, extended attributes, ZFS properties like compression settings — because the stream carries the actual ZFS metadata rather than a best-effort reconstruction of it.

The Manual Version, Understood First

Before reaching for the tools that automate this, it’s worth doing one full cycle by hand so the mental model is solid.

1
2
3
4
5
6

# take a snapshot on the source
zfs snapshot tank/documents@2023-12-01

# full send of that snapshot, piped to the destination over SSH
zfs send tank/documents@2023-12-01 | \
  ssh [email protected] zfs receive backup-pool/documents

That first send transfers the entire dataset — expensive, but it only happens once. Every subsequent backup is incremental, referencing the previous snapshot as the baseline:

1
2
3
4

# a week later: new snapshot, then an incremental send from the last one
zfs snapshot tank/documents@2023-12-08
zfs send -i tank/documents@2023-12-01 tank/documents@2023-12-08 | \
  ssh [email protected] zfs receive backup-pool/documents

The -i flag tells zfs send to transmit only the delta between the two named snapshots. The destination applies that delta to its existing @2023-12-01 copy and ends up with an exact @2023-12-08. Crucially, both snapshots have to still exist on the source for this to work — which is the first thing that catches people out.

Bookmarks: Keeping the Baseline Without Keeping the Snapshot

Retaining every snapshot you’ve ever sent, forever, just so you have a valid incremental baseline, defeats the purpose of snapshot retention policies. ZFS bookmarks solve this: a bookmark records the same point-in-time reference a snapshot does, for send purposes, but without holding onto the actual data blocks — it costs almost nothing to keep.

1
2
3
4
5
6

# create a bookmark from a snapshot, then the snapshot itself can be pruned
zfs bookmark tank/documents@2023-12-01 tank/documents#2023-12-01

# a later incremental send can use the bookmark as the baseline instead
zfs send -i tank/documents#2023-12-01 tank/documents@2023-12-08 | \
  ssh [email protected] zfs receive backup-pool/documents

This is the piece that lets you run an aggressive local snapshot retention policy (prune anything older than a week, say) while still being able to compute a valid incremental stream against a much older baseline whenever needed.

It’s worth understanding why this matters beyond the disk-space arithmetic. Without bookmarks, your local retention policy and your replication baseline are the same thing whether you want them to be or not — pruning a snapshot the destination hasn’t yet caught up to breaks the incremental chain, and the next send has to fall back to a full send of the whole dataset. On a slow upload link that’s not a minor inconvenience, it’s the difference between a nightly job that finishes before the household wakes up and one that’s still running at lunchtime. Bookmarks decouple those two concerns cleanly: keep a week of real snapshots locally for quick rollbacks, keep bookmarks going back months for replication baselines, and prune the snapshots on whatever schedule suits local recovery without ever touching the replication path’s ability to compute a delta.

sanoid and syncoid: The Part You Actually Run

Hand-typing zfs send commands does not scale to “every night, for every dataset, resuming correctly if last night’s run got interrupted”. sanoid handles local snapshot policy (creation and pruning by retention rules); syncoid, from the same project, wraps zfs send/receive with sensible defaults, automatic incremental detection, and resumability.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# /etc/sanoid/sanoid.conf
[tank/documents]
    use_template = production
    recursive = yes

[template_production]
    frequently = 0
    hourly = 36
    daily = 30
    monthly = 6
    yearly = 0
    autosnap = yes
    autoprune = yes

sanoid runs on a timer, taking and pruning snapshots per that policy — 36 hourly, 30 daily, 6 monthly, rolling off automatically. Then syncoid handles the off-site leg:

1
2

# run nightly via systemd timer or cron
syncoid --sendoptions="w" tank/documents [email protected]:backup-pool/documents

syncoid figures out the most recent common snapshot between source and destination automatically, sends the incremental from there, and — this is the feature that matters most for an off-site link with imperfect uptime — resumes a partial transfer if the connection drops mid-stream, rather than starting the whole incremental over. On a home upload connection, a multi-hour send failing at 90% and having to restart from zero is the difference between backups that actually complete and ones that never do.

The -w sendoption (--raw in older syntax, exposed as w here) sends encrypted datasets in their still-encrypted form when the source uses native ZFS encryption, meaning the off-site box can hold your backup without ever having the key to read it. That’s worth doing deliberately if “off-site” also means “a box I don’t fully control” — a relative’s house, a colocation you don’t administer day to day.

An Off-Site Strategy That Actually Works

The pattern I run: nightly syncoid push from the home pool to a small box at a relative’s house, reachable over a WireGuard tunnel (see The Tunnel Home for that side of it) so neither end needs a public IP or an open port. The receiving box uses native encryption with a key it never receives, courtesy of --raw/-w sends, so even if that box were compromised, the attacker gets encrypted blocks and nothing else.

1
2
3
4
5
6
7

#!/bin/bash
# /usr/local/bin/offsite-replicate.sh, run nightly via systemd timer
set -euo pipefail
DATASETS="tank/documents tank/photos tank/vm-storage"
for ds in $DATASETS; do
    syncoid --sendoptions="w" "$ds" "[email protected]:backup-pool/${ds#tank/}"
done

This covers one leg of a proper 3-2-1 setup — see The 3-2-1 Rule, Actually Implemented at Home for how it fits alongside a local-disk copy and a genuinely offline copy.

Choosing an Off-Site Destination

The destination doesn’t have to be exotic. A relative’s spare room, a colocated 1U box, a friend’s homelab with a reciprocal arrangement (you hold their backups, they hold yours), or a small VPS with attached block storage all work, because zfs receive doesn’t care what’s powering the box on the other end, only that it’s running ZFS and reachable over SSH. What matters more than the hosting arrangement is the trust model, which is exactly why the raw-send option gets its own emphasis above: a reciprocal arrangement with a friend is a genuinely good off-site strategy right up until you think honestly about whether you’d want that friend able to browse your documents dataset, and raw sends make that question moot by design rather than by politeness.

Bandwidth is the other practical constraint people underestimate. A first full send of a multi-terabyte pool over a typical home upload connection can take days, not hours — asymmetric consumer connections are usually the actual bottleneck, not ZFS. Budget for that reality up front: either seed the initial full send locally over a LAN or USB connection before the destination box ever leaves the house, or accept that the first sync is going to run in the background for the better part of a week and plan the send window accordingly. After that, incrementals are small enough that bandwidth stops being a meaningful concern for most home workloads, since you’re only ever pushing what actually changed since the last successful run.

Verifying a Backup Is Actually Restorable

A replication job that runs successfully every night for a year and has never once been test-restored is, at best, an untested assumption wearing the shape of a backup. The value of zfs send/receive here is that testing a restore is cheap enough to do routinely, unlike restoring a multi-terabyte tar archive from a file-level backup tool. Because the destination dataset is a genuine, mountable ZFS filesystem the whole time, you can clone a specific snapshot and actually look at the files without touching the live replicated dataset:

1
2
3
4
5
6
7
8
9

# on the destination, clone a specific snapshot to a throwaway dataset
zfs clone backup-pool/documents@2023-12-08 backup-pool/restore-test

# mount it, poke around, confirm the files you expect are actually there and intact
zfs set mountpoint=/mnt/restore-test backup-pool/restore-test
ls -la /mnt/restore-test

# done — discard the clone, the original replicated dataset was never touched
zfs destroy backup-pool/restore-test

I run this check monthly against a handful of specific files I know should be there, picked semi-randomly rather than always checking the same predictable path — the goal is catching a replication job that’s technically succeeding but silently missing something, which merely confirming the pipe works would never surface. A clone costs almost nothing in space (it only diverges from the snapshot as you write to it, which a read-only poke-around doesn’t) and takes seconds to create and destroy, so there’s no excuse for skipping this.

Troubleshooting

“cannot receive incremental stream: destination has been modified”. The destination dataset was written to (or a snapshot on it was deleted) since the last successful receive, breaking the chain of trust ZFS needs to apply an incremental cleanly. Keep the receiving side strictly read-only from anything except the replication process — never mount it read-write for casual use.

Resumed transfer keeps failing at the same point. Check disk space on the destination first; a partially-applied receive token still reserves space for the in-flight stream, and a nearly-full destination pool will fail resumption in a way that looks like a network problem but isn’t.

Send is far slower than expected on a first full send. Full sends of datasets containing large numbers of small, cold files aren’t actually faster than rsync in every case — they still have to walk the tree once for zfs send at the block level, and a slow upload link is a slow upload link regardless of protocol efficiency. Consider seeding the first full send locally (via an external disk carried to the destination) and only running incrementals over the network from then on.

zfs bookmark command not found or errors. Bookmarks require a reasonably recent OpenZFS version and the bookmarks and extensible_dataset pool features enabled. Check with zpool get all tank | grep feature@bookmarks and zpool upgrade tank if it shows disabled.

Raw (-w) receive fails with a key-related error. A raw-received encrypted dataset arrives locked, because the destination never got the key — that’s the entire point. zfs load-key will fail there deliberately; you only load the key on a box that’s actually supposed to be able to read the data. If you need to restore from the off-site copy back to a trusted machine, send it back with -w again and load the key only on that trusted destination.

What This Doesn’t Protect Against

Worth being explicit about the gap: zfs send/receive replicates snapshots, so its protection window is bounded by your snapshot frequency, same as any snapshot-based scheme. If sanoid takes hourly snapshots and syncoid replicates nightly, a mistake made and immediately compounded within that window (a script that both writes bad data and deletes the previous good snapshot before the next sync) can propagate to the off-site copy along with everything else. This is rare in practice but worth knowing about, which is why it’s worth keeping a reasonably deep retention window on the destination side too, deeper than a simple mirror of the source’s own pruning schedule, to give yourself a longer rollback horizon than the source pool alone provides.

Verdict

zfs send/receive, wrapped in sanoid/syncoid, is the closest thing to a free lunch in backup tooling: block-level efficiency, exact replicas including metadata, resumable transfers, and — with raw sends — an off-site copy the destination can’t even read. It requires ZFS on both ends, which is the real constraint; if your destination can’t run ZFS, you’re back to file-level tools. But if you’re already running ZFS at home, there’s very little reason not to have this pushing snapshots somewhere that isn’t your house, on a schedule you never have to think about again.