rsync Is Thirty Years Old and Still Unbeatable
Contents
Every backup tool I have tried in the last decade — and I have tried most of them, chasing some marginal improvement in speed or a nicer interface — eventually gets replaced by something newer, except one. Rsync, written by Andrew Tridgell in 1996 as part of his PhD work, still moves the bulk of my homelab’s backup traffic, because nothing since has meaningfully improved on the actual algorithmic idea at its centre. Newer tools wrap it in nicer interfaces, add encryption, add deduplication — Borg and Restic both do this well — but the core problem rsync solved, transferring only the parts of a file that actually changed, is still solved the same way it was thirty years ago, because the solution was already close to optimal.
The problem worth solving
Naive file synchronisation between two machines works by comparing whole files and re-transferring anything that differs, which is fine for small files and genuinely wasteful for large ones. A ten-gigabyte database dump that changes by a few megabytes between backups should not cost ten gigabytes of transfer to back up again, and before rsync, for most tools, it did exactly that. Tridgell’s insight was that you could identify which parts of a file changed without both machines already having a full copy to compare against, using a clever combination of two checksums computed over a rolling window.
How the rolling checksum actually works
The receiving side (the machine that already has an old copy of the file) splits its existing copy into fixed-size blocks and computes two checksums per block: a fast, weak rolling checksum (originally Adler-32-derived) and a slower, strong checksum (originally MD4, now typically a stronger hash) as a collision safeguard. It sends only this list of checksums to the sending side — not the file itself, just the checksums, which is a tiny amount of data compared to the file.
The sending side then slides a window of the same block size across its own copy of the file, byte by byte rather than block by block, computing the weak rolling checksum at every offset. The rolling part is the clever bit: because the weak checksum is designed so that sliding the window forward by one byte can be computed incrementally from the previous position’s checksum, rather than recomputed from scratch, checking every possible offset in the file is cheap rather than requiring a full recomputation per byte. When a weak checksum at some offset matches one in the receiver’s list, the sender checks the strong checksum to rule out a coincidental collision, and if that matches too, it knows this region of its file is identical to a block the receiver already has. Everything that doesn’t match a known block gets sent as literal new data; everything that does gets referenced by block number instead of transferred.
The practical result: two ten-gigabyte files differing by a few scattered megabytes will typically transfer only slightly more than those few megabytes, plus a modest overhead of checksums and block references, regardless of where in the file the differences fall or how the data shifted around. That “regardless of where” part is the genuinely hard bit — a naive approach only spots identical blocks if they land on the same file offset on both sides, and rsync’s rolling window finds matches anywhere, which is what makes it work well even when data has been inserted or deleted partway through a file, shifting everything after it.
What this actually looks like day to day
| |
-a (archive mode) preserves permissions, timestamps, symlinks and ownership, which matters enormously for anything you would actually want to restore correctly rather than just copy the bytes of. -z compresses data in transit, worth it over a slow link and mostly wasted CPU over a fast local network. --delete keeps the destination a true mirror by removing files on the far side that no longer exist on the source, which is exactly what you want for a mirror and exactly what you don’t want if you meant to keep historical versions — it is the single flag in that command most likely to surprise you the first time you run it against a directory you didn’t fully think through.
The trailing slash on the source path is the classic gotcha, and it changes behaviour completely: /srv/data/ (trailing slash) copies the contents of data into the destination directory, while /srv/data (no trailing slash) copies the data directory itself into the destination, nesting an extra level. Getting this backwards on a --delete run against the wrong nesting level is how people accidentally delete an entire backup tree they meant to preserve, and it is worth testing with --dry-run before running any command combining --delete with a source path you have not triple-checked.
Where rsync fits next to a real backup tool
Rsync’s delta algorithm is the engine underneath several more complete backup tools, and understanding rsync well is what makes those tools’ behaviour predictable rather than magical. Rsync itself does not deduplicate across files, does not encrypt at rest, and does not version history beyond whatever --link-dest hard-link trickery you build yourself — it transfers a delta of one file against another copy of that same file, full stop. Tools that add encryption, cross-file deduplication and proper snapshot history on top of a similar delta idea are solving a different, larger problem, and reaching for rsync directly still makes sense for straightforward mirroring — a NAS-to-NAS sync, a config directory shipped to a secondary host, a website’s static output pushed to a server — where you want the delta-transfer efficiency without the overhead of a full backup tool’s metadata and retention model.
Two transports, and why the choice matters
Rsync can run over two genuinely different transports, and picking the wrong one for the job is a common source of confusion. Running it over SSH, as in the example above, wraps every byte of the rsync protocol in an encrypted tunnel and reuses whatever authentication you already have set up for that host — the simplest and most common choice for anything crossing a network you don’t fully control, including a home broadband connection to an off-site backup target. Running it against a dedicated rsync daemon (rsync://host/module syntax, backed by rsyncd.conf on the receiving side) skips SSH’s encryption and authentication overhead entirely, trading it for the daemon’s own simpler access controls, which is a reasonable choice on a trusted local network where the encryption overhead buys you nothing against a threat model that doesn’t include anyone sniffing your own LAN.
The daemon mode also unlocks a few module-level conveniences SSH transport doesn’t give you as cleanly — per-module read-only flags, connection limits, and anonymous access for something like a public mirror — which is why you still see rsync:// mirrors serving Linux distribution ISOs rather than everyone reaching for SSH transport for that use case. For a homelab backup job talking to a machine you control and trust, either transport works; SSH is the safer default the moment any part of the path crosses a network you don’t fully own.
Throttling and scheduling it politely
A large rsync job over a home connection’s upload bandwidth will happily consume all of it, which is a problem the moment something else on the network — a video call, another household member’s traffic — needs that bandwidth at the same time. --bwlimit=5000 (in KB/s) caps the transfer rate rather than letting rsync saturate the link, and it is worth setting deliberately on any job that runs during hours the connection is doing other things, rather than discovering the contention the first time a backup job coincides with something time-sensitive.
Scheduling matters as much as throttling for the same reason. A nightly rsync job timed for the small hours avoids the contention question entirely for most households, provided the destination host and the network path between the two are actually available at that hour, which is worth verifying rather than assuming — a laptop that suspends overnight, or a NAS that spins down its drives on a schedule that doesn’t line up with the backup window, will silently turn a scheduled job into a job that fails quietly every night until someone happens to check.
Troubleshooting the errors that actually happen
Permission errors on the destination almost always mean the receiving user lacks write access to the target path, and the fix is either running the destination-side process as a user with the right permissions or, for a proper backup target, setting up a dedicated rsync daemon or restricted SSH user rather than reflexively running everything as root over SSH, which works but is a needlessly broad grant of access for what should be a narrowly scoped backup job.
A sync that appears to hang partway through a large file transfer, rather than failing outright, is worth checking against network stability before assuming rsync itself has a problem — rsync has no built-in resume for a mid-transfer interruption within a single file the way a segmented downloader would, so a flaky link partway through a large file means starting that file’s transfer over, though --partial at least keeps the incomplete data on the receiving side so a subsequent run can use it as a starting point for the delta comparison rather than discarding it entirely.
Unexpectedly slow transfers on what should be a fast local network are frequently a compression tax rather than a genuine bottleneck: -z costs CPU time compressing data that a Gigabit or faster local link would have carried uncompressed just as quickly, so dropping the flag on local transfers and reserving it for genuinely bandwidth-constrained links (a remote backup destination over a home broadband upload, say) often speeds things up rather than the intuitive assumption that compression always helps.
Files that seem to sync repeatedly despite no apparent changes are usually a timestamp precision mismatch between filesystems — some filesystems store modification times to the second, others to the nanosecond, and comparing across that boundary can make rsync believe a file changed when only its stored precision did. Adding --size-only for cases where you trust file size alone as the change indicator, or --checksum to force an actual content comparison rather than trusting timestamps at all, resolves it at the cost of a slower comparison pass.
A destination that fills up mid-transfer, particularly on a --delete mirror job where old data is removed after the new data lands rather than before, deserves specific attention: rsync by default writes new file contents to a temporary file alongside the real target and renames it into place only on successful completion, which protects you from a half-written file replacing a good one, but does mean the destination briefly needs enough free space for both the old and new copies of anything large that changed. A destination running consistently close to full is one interrupted transfer away from a job that fails halfway and leaves temporary files scattered across the target, which --delete-during versus --delete-after behaviour affects in ways worth reading the manual page for rather than guessing at when free space is already tight.
Is it still worth reaching for
Completely, and specifically for the job it was built for: efficiently mirroring or syncing files where bandwidth or time matters and a full backup tool’s overhead is not needed. Pair it with a proper scheduling mechanism — cron remains the honest choice for this, and a dead man’s switch watching that the job actually ran closes the gap between “the job is scheduled” and “the job actually completed successfully last night,” which are not the same claim and conflating them is how silent backup failures happen. For genuine bulk cold storage sitting behind that sync job, I have found a dedicated cold-storage tier a more honest fit than trying to make a primary NAS also serve as the archive tier. Thirty years on, rsync remains the tool I reach for first, because the newer options are mostly rsync’s idea with more scaffolding around it, and that scaffolding is not always what a given job needs.
Related Content
