Litestream: Streaming SQLite Backups to Object Storage

Count the SQLite databases in your homelab and the number will surprise you. That bookmark manager, the RSS reader, the photo app, the link shortener, the home-automation hub, half the small Go and Rust services people self-host — an enormous amount of them store their entire state in a single .db file and never mention it. SQLite is the most deployed database engine on the planet precisely because it is invisible. Which means an enormous amount of homelab data is sitting in files that most people are not backing up correctly, if at all.

“Correctly” is the operative word, because copying a live SQLite file is where people quietly corrupt their own backups. Litestream fixes this by continuously streaming the database’s changes to object storage as they happen, giving you point-in-time restore with seconds of potential data loss rather than a stale nightly cp that might be inconsistent. This post covers why the naive approach fails, how Litestream works, a real config, and the handful of gotchas that will otherwise ruin your afternoon.

Why you cannot just copy the file

The obvious backup for a single-file database is to copy the file. For SQLite this is a trap, and it is worth understanding precisely why so the fix makes sense.

A live SQLite database is often mid-transaction. In the default rollback-journal mode, and even more so in the Write-Ahead Logging (WAL) mode most serious apps enable, the “current” state of the database is spread across the main .db file plus a -wal file holding committed changes that have not yet been folded back into the main file. Copy just the .db while the app is writing and you can capture a torn state: pages from two different transactions, a header that disagrees with the body, changes that live only in the WAL you did not copy. The copy might open. It might even mostly work. Then one day it fails an integrity check on a table you needed, and you discover your “backups” were subtly broken for months. This is the database-shaped version of the lesson in snapshots are not backups: a copy that was never guaranteed consistent is not a copy you can rely on.

You can do it properly with the SQLite backup API or VACUUM INTO, which produce a consistent snapshot, but those are point-in-time — run one nightly and you can still lose a full day of writes if the disk dies at teatime. For a database that changes all day, you want something continuous.

What Litestream actually does

Litestream is a small single-binary daemon that sits beside your application and replicates a SQLite database by tailing its WAL. Here is the mechanism, because it explains every one of the gotchas later.

SQLite in WAL mode appends every committed transaction to the -wal file as a series of frames, and periodically “checkpoints” — folds those frames back into the main database and truncates the WAL. Litestream opens a long-lived read connection to your database, which holds the WAL open and prevents SQLite from checkpointing away frames Litestream has not yet copied. It watches the WAL, ships each new frame up to object storage, and lets the checkpoint proceed once the frames are safely replicated. What lands in your bucket is a continuous stream of every change, from which Litestream can reconstruct the database at any point in the retained history.

The result is a replica in S3-compatible storage (or Azure Blob, Google Cloud Storage, or SFTP) that trails your live database by seconds, with point-in-time restore across the retention window. It is a disaster-recovery tool: it gives you an off-machine, continuously updated copy you restore from when the primary is gone. It does not serve live reads or run a hot standby, so treat it as the backup leg for your SQLite apps and nothing more.

A real configuration

Install the binary (Litestream ships .deb and .rpm packages, or a static binary), then describe your databases in /etc/litestream.yml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# /etc/litestream.yml
dbs:
  - path: /var/lib/myapp/app.db
    replicas:
      - type: s3
        bucket: homelab-litestream
        path: myapp/app.db
        endpoint: s3.eu-central-1.amazonaws.com
        region: eu-central-1
        # How far back point-in-time restore reaches
        retention: 168h
        # Take a full snapshot periodically so restores don't
        # have to replay the entire WAL history from scratch
        snapshot-interval: 24h

  - path: /var/lib/otherapp/data.db
    replicas:
      - type: s3
        bucket: homelab-litestream
        path: otherapp/data.db
        endpoint: s3.eu-central-1.amazonaws.com
        region: eu-central-1

Credentials go in the environment, never in the file:

1
2

export LITESTREAM_ACCESS_KEY_ID="..."
export LITESTREAM_SECRET_ACCESS_KEY="..."

The same config points at any S3-compatible endpoint — Backblaze B2, MinIO, Wasabi, a self-hosted object store — by changing endpoint and bucket. Because everything you care about ends up in a bucket, the same off-site object storage you might already use for restic and rclone offsite backups does double duty here, and Litestream becomes the off-site leg of the 3-2-1 rule for every SQLite app you run.

Run the daemon, which replicates all configured databases:

1

litestream replicate

Running it under systemd

You want this running whenever the machine is up, restarting on failure, with credentials loaded from a locked-down environment file. Litestream’s package ships a service, but here is the shape so you understand it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# /etc/systemd/system/litestream.service
[Unit]
Description=Litestream continuous SQLite replication
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
EnvironmentFile=/etc/litestream/litestream.env
ExecStart=/usr/bin/litestream replicate
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Put the two LITESTREAM_* keys in /etc/litestream/litestream.env, chmod 600 and owned by root. Enable it with systemctl enable --now litestream, and confirm frames are actually flowing with litestream databases and litestream generations /var/lib/myapp/app.db, which shows the replicated generations and how far behind they are.

Watch the lag and the bill

A streaming backup that silently stopped streaming is the cruellest failure mode, because everything looks fine until the day you restore and discover the replica froze three weeks ago. Two things want an eye on them once this is running. The first is replication lag: litestream generations reports the position of the replica against the live database, and if that gap stops shrinking, replication has stalled and you want an alert. The pragmatic homelab move is to have a small nightly check compare the local database’s modification time against the replica generation and ping a healthcheck URL on success, so silence — a stalled daemon, dead credentials, a full bucket — is what raises the alarm.

The second is cost, and it is genuinely small. Litestream ships WAL frames, which for most homelab apps means a trickle of writes and a full snapshot every day or so. On an S3-compatible store you are paying for a few gigabytes of storage and a modest number of PUT requests, which for a handful of small databases lands in the pennies-to-low-single-digits per month range. Set a sane retention (a week is plenty for most apps) so old generations expire and the bucket does not creep upward forever. If you run many databases, watch the request count rather than the byte count — on chatty write workloads it is the per-request charges that add up first, well before storage volume becomes the dominant line on the bill.

Restoring — the part that must actually work

A backup you have never restored is a rumour. Litestream restores are refreshingly simple: point it at the replica and an output path, and it rebuilds the database by applying the WAL stream on top of the most recent snapshot.

1
2
3

# Restore the latest state to a new file
litestream restore -o /var/lib/myapp/app-restored.db \
  -config /etc/litestream.yml /var/lib/myapp/app.db

To roll back to a specific moment — say just before a bad migration ran at 14:05 — restore to a timestamp:

1
2
3

litestream restore -o /tmp/app-1400.db \
  -timestamp 2023-11-20T14:00:00Z \
  -config /etc/litestream.yml /var/lib/myapp/app.db

The workflow for a real recovery is: stop the app, restore to a fresh file, run PRAGMA integrity_check; on it with the sqlite3 CLI, then move it into place and start the app. Always restore to a new path and verify before overwriting the live file, so a botched restore never destroys the working database. Rehearse this once a quarter against a throwaway copy so that the day you need it, your hands already know the commands.

Troubleshooting

“database is not in WAL mode” or replication does nothing. Litestream requires WAL journal mode. Most apps set it, but if yours does not, enable it once with sqlite3 app.db 'PRAGMA journal_mode=WAL;'. Without WAL there is no frame stream for Litestream to tail.

The -wal file grows without bound and the disk fills. Litestream holds a read lock to stop SQLite checkpointing away un-replicated frames. If replication to your bucket stalls — bad credentials, network down, bucket unreachable — frames pile up in the WAL because they cannot be checkpointed until they are safely uploaded. Check journalctl -u litestream for upload errors, fix the bucket connectivity, and the WAL drains as replication catches up. Monitor free disk on databases with heavy write volume.

Two processes replicating one database. Run exactly one Litestream instance per database. A second litestream replicate against the same file, or your app doing its own aggressive checkpointing, fights Litestream for the WAL and produces gaps. One daemon owns replication for a given database.

Restore succeeds but the app complains the database is corrupt. Almost always you copied the live file over a running app, or restored while the app still held the database open. Stop the app first, restore to a new path, integrity-check, then swap.

Litestream on a network filesystem behaves oddly. SQLite’s locking is unreliable over NFS and SMB, and Litestream inherits that. Replicate databases that live on local disk. If an app’s data is on a network share, that is a different problem to solve before Litestream enters the picture.

Restores are slow and pull huge amounts from the bucket. If snapshot-interval is too long, a restore has to replay a vast WAL history since the last full snapshot. Shorten the snapshot interval so restores start from a recent full copy and replay less.

Verdict: is it worth it, and for whom?

If you self-host anything backed by SQLite — and you almost certainly host more than you realise — Litestream is close to essential and costs you a single binary, a few lines of YAML, and pennies of object storage. It closes the specific, nasty gap where a nightly file copy of a busy database is either stale or quietly corrupt, and it gives you point-in-time restore measured in seconds of data loss instead of a day. For small services where the entire state is one .db file, it is the most reassurance per minute of setup I know of.

It is not a general backup system and does not pretend to be. It replicates SQLite databases and only SQLite databases; your Postgres instances, your media library, your config files and everything else still need their own strategy, and Litestream sits alongside those rather than replacing them. It is a disaster-recovery replica that you restore from when the primary is gone; it does not serve live reads or run a hot standby, so genuine high availability or live read replicas for SQLite call for a different tool and a different article. For the specific and extremely common case of “I have a pile of small apps each guarding their data in one SQLite file and I would like to never lose that data,” set up Litestream this weekend, then prove it by restoring one of them to a scratch file before you trust it. The databases running half your homelab deserve better than a cp you never tested.