Calibre-Web-Automated: Fixing the Ebook Workflow

My ebook library spent about six years in a state I would describe as “technically working”. Calibre — the desktop application, the one Kovid Goyal has been maintaining since 2006 — was the actual brain of the operation. It held the metadata database, it did the conversions, it talked to my e-reader over USB. And it lived on a laptop that was closed and in a drawer roughly ninety per cent of the time.

The result was a two-tier system. Books I had properly imported were beautiful: correct author, correct series index, a cover, a sensible EPUB. Books I had acquired since the last time I opened the laptop lived in a folder called _toimport that eventually contained 340 files with names like dune-1.epub and 9780441013593.azw3. The library was fine. The pipeline into the library was a manual chore I performed twice a year, badly.

Calibre-Web solved half of this in 2017 by putting a web front end on top of an existing Calibre metadata.db. It reads the library, gives you a browsable catalogue, OPDS feeds, a send-to-Kindle button and user accounts. What it deliberately does not do is manage the library. It is a reader for a database that something else maintains. If you wanted a new book in there, you still opened the laptop.

Calibre-Web-Automated is a fork that closes that gap. It keeps everything Calibre-Web does and adds the part I had been faking with cron jobs: a watched folder that ingests, converts, de-DRMs (if you have the tools), fixes metadata and files the result — with no desktop application involved. I have been running it since it was fairly young, and the drawer laptop has not been opened since.

Why the ingest folder is the whole point

The insight behind CWA is small and, once you see it, obvious. Calibre already ships a complete command-line toolchain: calibredb to manipulate the library database, ebook-convert to transform formats, ebook-meta to read and write metadata. These binaries are the same code the GUI drives. Nothing about them requires a desktop.

So the fork does this: it runs the Calibre binaries headless inside the container, watches a directory with inotify, and when a file lands, it runs the pipeline. Convert to your target format if the incoming format is on the conversion list. Enforce the metadata you have configured. Add it to the library with calibredb add. Delete the source file. Trigger a Calibre-Web re-scan so the book appears in the catalogue.

The behavioural change is disproportionate to the technical change. When filing a book costs one drag-and-drop into a network share, books get filed. When it costs “open laptop, wait for Calibre to start, import, fix author name, convert, eject” — they do not. I went from a 340-file backlog to zero within a week, and the backlog has not come back, because there is no longer a step where a human has to remember something.

This is the same lesson I keep relearning across the rack. The value of automation is rarely the seconds saved; it is that the task stops requiring a decision. I made roughly the same argument about scanned documents in Paperless-ngx: A Paperless Office That Actually Works, and it applies identically here.

What you actually get over vanilla Calibre-Web

Worth being precise, because the fork’s README is enthusiastic and the practical differences are narrower than they first appear.

Auto-ingest. The headline. Drop files in /cwa-book-ingest, they get processed and vanish. Directories are walked recursively, so you can dump an unzipped archive in and walk away.

Auto-convert. You nominate a target format (EPUB in my case, because Kobo). Anything arriving as AZW3, MOBI, PDF, FB2 or whatever else gets run through ebook-convert. You can also nominate formats to leave alone — I keep PDFs as PDFs, because converting a PDF to EPUB produces something that technically exists.

Metadata enforcement. You can force the library’s metadata onto the book files themselves, so the EPUB you send to a device carries the author name you fixed in the web UI rather than the one the publisher typed at 4pm on a Friday.

Automatic backup of ingested originals. Before processing, the original file is copied to a backup directory. This matters more than it sounds — see the troubleshooting section.

A one-shot library convert tool. Run it once against an existing library to normalise every book to your target format. On a library of 1,200 titles this took my mini PC about four hours and pegged all cores. Do it overnight.

What it does not give you is DRM removal. CWA supports the DeDRM plugin if you install it, and the legality of doing so depends entirely on where you live and what you bought. I am not your solicitor.

The compose file

Here is the stack, genericised. It assumes you already have a Calibre library directory containing a metadata.db — if you do not, CWA will create an empty one on first run.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

services:
  calibre-web-automated:
    image: crocodilestick/calibre-web-automated:latest
    container_name: calibre-web-automated
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/Copenhagen
      # Optional: force the ingest pipeline to target EPUB
      - NETWORK_SHARE_MODE=false
    volumes:
      # The Calibre library itself: must contain metadata.db
      - /srv/books/library:/calibre-library
      # Drop zone. Anything landing here gets ingested and removed.
      - /srv/books/ingest:/cwa-book-ingest
      # Config, users, CWA's own settings database
      - /srv/appdata/cwa/config:/config
    ports:
      - "8083:8083"
    restart: unless-stopped

Three volumes, one port, no database server. The whole thing is SQLite underneath, which for a single-digit number of concurrent readers is entirely adequate — an argument I have made at more length in SQLite Is a Production Database (For Your Homelab, Anyway).

On first boot, browse to http://mylab.local:8083, log in with the default admin / admin123, and change that immediately. The database location it asks for is /calibre-library — the container path, not the host path. This trips up roughly everyone once.

Then, in Settings → Calibre-Web-Automated, set your target format and your conversion ignore list. My configuration:

1
2
3
4

Target format:        EPUB
Ignored formats:      pdf, cbz, cbr
Auto-backup ingest:   enabled
Enforce metadata:     enabled

Test it with one book before you fire a hundred at it:

1
2
3
4
5
6
7
8

$ cp ~/Downloads/some-book.azw3 /srv/books/ingest/
$ docker logs -f calibre-web-automated
[cwa-ingest] New file detected: /cwa-book-ingest/some-book.azw3
[cwa-ingest] Converting azw3 -> epub
[cwa-convert] ebook-convert exited 0 in 11.4s
[cwa-ingest] Adding to library: "Some Book" by A. Author
[cwa-ingest] calibredb add ok (id 1247)
[cwa-ingest] Removing source file

If those lines appear, you are done, and the rest of your ebook life is a network share.

Running it behind something sensible

Port 8083 on the LAN is fine for a week. After that, put it behind a reverse proxy so you get TLS and a name a human can type. My setup is the boring one:

1
2
3

books.mylab.local {
    reverse_proxy 192.168.1.50:8083
}

Calibre-Web supports reverse-proxy header authentication, which lets your SSO layer do the login and pass the username through. It works, and I use it. Be aware of the standard footgun: if header auth is enabled and the app is reachable without going through the proxy, anyone who can reach port 8083 directly can assert any username they like. Bind the container’s port to localhost, or firewall it, or both. The same caveat applies to every app that trusts a header, and I laboured the point in Authelia vs Authentik: Choosing a Self-Hosted SSO.

The other thing worth wiring up on day one is the OPDS feed, at /opds. Every serious reader app on a phone or tablet speaks OPDS, and it means the catalogue is browsable from the sofa without CWA needing to know what device you are on.

Troubleshooting: the four things that will actually go wrong

Books ingest, then disappear entirely. The source file is deleted after processing, and if the calibredb add step failed silently, you have lost it. This is why the auto-backup setting exists and why you should turn it on before your first bulk import. The backup directory lives under /config; check it there. I lost eleven books learning this, and they were the eleven with the most awkward metadata, which is the sort of joke the universe finds funny.

Permission denied on the library directory. The classic. PUID/PGID in the compose file must match the owner of /srv/books/library on the host, and the ingest folder must be writable by that user too, since CWA deletes from it. Check with:

1
2
3

$ stat -c '%u %g %n' /srv/books/library /srv/books/ingest
1000 1000 /srv/books/library
1000 1000 /srv/books/ingest

If your NAS share is mounted with a fixed uid=/gid= in fstab, that number is what goes in the compose file, whatever /etc/passwd says.

Conversions hang forever on huge PDFs. ebook-convert on a 400MB scanned PDF will happily consume a core for half an hour and produce garbage. Put pdf in the ignore list. If you genuinely need text out of scanned PDFs, that is an OCR problem, and OCR belongs in a different tool.

Metadata enforcement overwrites something you wanted. Enforcement is one-directional: library database wins, file loses. If you have hand-edited an EPUB’s internal metadata and expect that to survive, it will not. Fix metadata in the web UI and let it propagate downward. Fighting the direction of flow ends badly.

The container restarts in a loop after an image update. CWA is a fast-moving fork and latest has bitten me exactly once, when a schema migration on the CWA settings database went sideways. Pin the tag. This is a hill I will keep dying on and I made the full case for it in Container Image Housekeeping: Pruning, Pinning, and Not Running :latest in Production.

Getting books onto the actual device

The library is only useful if books leave it. There are three routes out, and they suit different readers.

OPDS is the one I use most. Point KOReader, Moon+ Reader or Aldiko at https://books.mylab.local/opds, authenticate once, and the whole catalogue is browsable from the device with downloads on tap. It is a 2010-era XML standard doing exactly what it was designed for, with no cloud in the path.

Send-to-Kindle posts the book to an Amazon address by email. It works, and it requires handing your library to an SMTP server and Amazon’s ingestion pipeline, which is a peculiar way to treat a file you already own. CWA has a built-in SMTP configuration for it under Settings → Email Server.

Kobo Sync is the interesting one. Calibre-Web can impersonate the Kobo store API, so the e-reader’s own native sync — the one built into the firmware — pulls from your server instead. You paste a generated URL into the device’s config, and from then on the Kobo believes your homelab is Rakuten. Reading position and progress sync back. It is delightfully cheeky and it has survived several firmware updates on my device, though it is precisely the kind of arrangement that breaks the day a vendor changes an endpoint.

1
2
3

$ # Enable per-user Kobo sync token in the web UI, then verify the endpoint answers
$ curl -s -o /dev/null -w '%{http_code}\n' https://books.mylab.local/kobo/<token>/v1/library/sync
200

Pick one and commit. Running all three means three sets of read-position state disagreeing with each other, and the reconciliation story for that is “there isn’t one”.

Back it up, because the database is the library

The .epub files on disk are replaceable. The metadata.db beside them — every author correction, every series index, every read/unread flag, every custom column — is the part that represents actual human effort. Treat it as the crown jewel.

SQLite and rsync-style backups have a well-known interaction problem: copying a database file mid-write gives you a corrupted copy. Use the proper tool:

1
2

$ sqlite3 /srv/books/library/metadata.db ".backup '/srv/backup/metadata-$(date +%F).db'"
$ restic backup /srv/backup /srv/books/library --exclude '*.db-wal' --exclude '*.db-shm'

.backup takes a consistent snapshot while the database is live. Run it from cron before your nightly backup job, and check the restore occasionally — a habit I have banged on about in Testing Your Backups: The Restore You Never Rehearsed and will keep banging on about, because untested backups have a failure rate that rounds to one hundred per cent at the exact moment you need them.

The honest verdict

Calibre-Web-Automated is a fork of a project that wraps a desktop application’s CLI. That sentence should tell you what the risk profile is. It depends on Calibre-Web upstream continuing to exist, on the Calibre binaries keeping their interfaces, and on one maintainer’s enthusiasm. The fork is young. The documentation assumes you already know what Calibre is. Updates arrive quickly and occasionally arrive sideways.

Against that: it does the one thing that made my library rot, and it does it well enough that I stopped thinking about the problem. That is a very high bar to clear, and most self-hosted software never clears it.

Who should run this: anyone with more than a few hundred ebooks, a device that wants a specific format, and a folder somewhere full of files they mean to import one day. If that folder exists, this software is aimed directly at you.

Who should not: anyone with fifty books and a Kindle. Calibre on a desktop, twice a year, is genuinely fine. The whole value here is amortising a chore across a large library, and below some threshold the chore is smaller than the container.

I am in the first camp, obviously. The laptop is still in the drawer. As far as I am concerned that is the metric.