ZFS Send/Receive: Time Travel for a Home Filesystem
Ship whole filesystem states across the network, one snapshot at a time
Contents
The first time I used ZFS send to copy a 400 GB dataset onto a second machine and then, an hour later, shipped only the 60 MB that had changed since, I understood why people who run ZFS get slightly evangelical about it. Every other copy tool I had used treated a filesystem as a bag of files to be walked, compared and diffed. ZFS treats it as a sequence of states, each one a perfect frozen photograph of the whole dataset, and it can hand you the exact difference between any two of those photographs as a byte stream you can pipe wherever you like. That difference is the whole trick, and once it clicks you start thinking about home storage as something you can rewind.
This is a guide to zfs send and zfs receive for a home setup: what they actually do, why they beat file-level copies for whole-filesystem replication, and the specific errors that will trip you up the first week. I will assume you already have a pool and at least one dataset. If you are still deciding on disk layout, the mirrors-versus-parity trade-offs are worth reading first, because send/receive sits on top of whatever pool you build.
Why snapshots make this cheap
ZFS is copy-on-write. When you modify a file, the old blocks are never overwritten in place; new blocks are written elsewhere and the metadata is updated to point at them. A snapshot simply tells the pool to keep the old block pointers around instead of freeing them. Taking a snapshot is therefore close to instantaneous and costs almost nothing at creation time — it only starts consuming space as the live data diverges from the frozen version and the old blocks it references can no longer be released.
That design has a lovely consequence for replication. Because ZFS already knows exactly which blocks changed between snapshot A and snapshot B, it can generate a stream containing only those blocks. There is no directory walk, no per-file stat, no checksum comparison across millions of files. A dataset with a million tiny files replicates just as efficiently as one with a single huge file, because the unit of work is the changed block, decided at the pool level. rsync has to open and compare; ZFS already knows.
Take a snapshot and list it:
| |
The @ separates the dataset from the snapshot name. Snapshots are read-only by definition, and you can browse one directly under the hidden .zfs/snapshot/<name>/ directory inside the dataset’s mountpoint — handy for grabbing a single file you deleted this morning without any restore ceremony at all.
The full send
A send stream is just data on stdout. A receive reads data on stdin. That Unix-y design is why send/receive composes with SSH, mbuffer, gzip and anything else that speaks pipes. The most basic replication is a full send of one snapshot into a new dataset on another pool:
| |
Across the network, you wrap the receive side in SSH:
| |
The receiving dataset is created for you. After this runs, backup/home on the remote box is a bit-for-bit copy of tank/home as of that snapshot, including every property ZFS knows about. The -v flag prints progress and a final summary so you can confirm how much moved.
That full send is the expensive one — it ships everything. You do it once. From then on you send differences.
The incremental send, where the magic is
Take a second snapshot later, then send only the delta between the two:
| |
Only the blocks that changed between the 12th and the 13th cross the wire. On a dataset that barely moved overnight, that might be a few megabytes regardless of how large the dataset is. This is the part that makes ZFS replication genuinely feel like time travel: the destination now holds both snapshots, and you can roll it back and forth between them at will.
There is a stricter variant, -I (capital), which sends every intermediate snapshot between the two named ones in a single stream, so the destination ends up with the full snapshot history rather than only the endpoints. Lowercase -i sends just the two endpoints. For a nightly replication job you usually want -I so the backup keeps the same retention history as the source.
One rule governs every incremental: the destination must still hold the snapshot you are sending from. If backup/home no longer has @2023-07-12, ZFS cannot compute the delta and the receive fails. This is why replication scripts pin a “last successfully replicated” snapshot on both sides and only prune older ones once a newer common snapshot exists.
Doing it properly with automation
You can script the snapshot-and-send loop yourself, but the sanoid/syncoid pair from Jim Salter has been the home-lab default for years and handles the fiddly bits — retention policies, finding the most recent common snapshot, resuming, and using mbuffer for throughput. A syncoid one-liner replaces the whole manual dance:
| |
syncoid works out the last common snapshot, sends the incremental, and manages the “keep from-snapshot around” bookkeeping for you. Pair it with sanoid on the source to take and expire snapshots on a schedule (hourly, daily, monthly) and you have a replication system in about fifteen lines of config. Drive it from a systemd timer or cron and it runs untouched for months.
Snapshots and replication are not a complete backup strategy on their own, though. Replicating a dataset copies your mistakes just as faithfully as your good data — delete a directory and the deletion replicates too. That is the whole argument of why snapshots are not backups, and it is why send/receive is one leg of a wider plan rather than the whole thing. Slotting it into the 3-2-1 rule properly implemented is what turns “I have a second copy” into “I can survive a fire”.
Resume tokens: surviving a dropped connection
A large full send over a home uplink can take hours, and home links drop. In the old days a broken pipe meant starting the multi-hundred-gigabyte transfer from zero, which was soul-destroying. Modern ZFS fixes this with resumable receive. Add -s to the receive side:
| |
If that transfer dies halfway, the partially-received dataset holds a resume token. Read it on the destination and use it to pick up exactly where you left off:
| |
The token encodes precisely which blocks already arrived, so the resume ships only what is missing. For seeding a large off-site copy over a domestic connection this feature alone is worth the switch to ZFS.
Raw encrypted sends
If your source dataset uses ZFS native encryption, you can replicate it without the destination ever holding the key, using a raw send:
| |
The -w (raw) flag sends the encrypted blocks as-is. The backup host stores ciphertext it cannot read; the data only decrypts if someone loads the key on the destination, which you never do there. That is exactly what you want for an off-site target you do not fully control — the same threat model that makes encrypted Restic backups to an untrusted remote safe. The backup server is a dumb vault holding opaque blocks.
Troubleshooting: the errors you will actually hit
cannot receive incremental stream: destination has been modified since most recent snapshot. Something wrote to the destination dataset after the last receive — often because it was mounted read-write and a process touched it, or you browsed and something updated access times. Set the destination to readonly=on (zfs set readonly=on backup/home) so nothing but zfs recv ever changes it. To recover the current mismatch, roll the destination back to its latest snapshot with zfs rollback before receiving again.
cannot receive incremental stream: most recent snapshot of X does not match incremental source. The from-snapshot you are sending no longer exists on the destination, or the two sides pruned themselves out of a common snapshot. List snapshots on both ends (zfs list -t snapshot), find the newest name they share, and send incrementally from that. If they share nothing, you are back to a full send.
The transfer stalls or crawls over SSH. SSH’s own buffering plus a bursty ZFS stream can throttle throughput badly. Insert mbuffer on both ends to smooth it out — syncoid does this automatically, and it routinely doubles or triples real-world speed on a home link. A large socket buffer stops the pipeline stalling every time the disk pauses.
A resumed send says the token is invalid. Resume tokens are tied to a specific partial receive. If you destroyed the partial dataset, ran a different send in between, or the source snapshot was deleted, the token is dead — clear it with zfs recv -A backup/media on the destination to abort the partial state, then start the send fresh.
Out of space on the receiver mid-stream. ZFS reserves space for metadata and refuses to fill a pool completely. A receive that dies with out of space usually means the destination pool is genuinely too small for the dataset plus its snapshot history — prune old snapshots on the destination, or the pool needs more disks. Replicating snapshot history means the backup can need more space than the live source, not less, because it retains states the source has already expired.
zfs recv complains the dataset already exists. A full send into an existing dataset needs -F to force a rollback of the destination to match the stream, which discards anything on the destination newer than the stream. Use -F deliberately, because it throws away destination-side changes.
Verdict: is it worth it, and for whom?
If you already run ZFS, send/receive is not optional — it is the single best reason to have chosen ZFS in the first place, and you should set up replication the same week you build the pool. Block-level incrementals over SSH, resumable transfers, raw encrypted sends and instantaneous snapshots together give you a home replication story that file-level tools cannot match, and the whole thing automates down to a syncoid line on a timer.
If you are not on ZFS, this alone is a fair reason to consider it for your bulk storage, though it is a real commitment: ZFS wants RAM, rewards a bit of study, and does not love mismatched or shrinking pools. For someone with a NAS holding data they would hate to lose and a second box or a friend’s server to replicate to, the payoff is enormous. For a single laptop with a handful of documents, a simpler file backup is the saner choice and you will never miss the snapshots. Build the pool, take snapshots on a schedule, replicate them off the box, and remember that replication is one copy in a wider plan — the deletion you replicate at 2 a.m. is still gone from both machines by breakfast unless something else is keeping older states safe.
Related Content
