Watchtower Considered Harmful: Safer Container Updates

At 04:12 one morning a while back, a piece of software I had installed and forgotten about pulled a new image tagged latest, stopped my PostgreSQL container, deleted it, created a new one from an image containing a major version bump, and started it. PostgreSQL 16 looked at a data directory written by PostgreSQL 15, said — correctly and politely — that it could not proceed, and exited. Everything that depended on that database went down behind it.

The old image had already been pruned, because I had enabled cleanup, because leaving dangling images around seemed untidy. So the rollback path was to work out which digest I had been running from a log line, hope it was still in the registry, and pull it back. It was. I got lucky. The whole episode cost me two hours and taught me something I should have known: an automation that changes production and cannot undo the change is a bet, and you are placing it while asleep.

The software was Watchtower. I do not think Watchtower is badly written. I think the job it does is one that cannot be done safely with the information it has.

What Watchtower actually does

Watchtower is a container that mounts the Docker socket, reads the list of running containers, and for each one asks the registry whether the tag it was started from now points at a different digest. If it does, Watchtower pulls the new image, stops the container, removes it, and recreates it with the same configuration.

That is a genuinely useful primitive and the implementation is competent. It handles the config-copying carefully, it supports Shoutrrr notifications, and it has a --run-once mode that does exactly what you would want.

The problem lives in the gap between “the tag moved” and “I should deploy this”.

The four things it cannot know

It cannot know what changed. A moving tag like latest or stable carries no semantics. The digest behind it changed. That could be a rebuild against a patched base image, a bug fix, a feature, a breaking configuration change or a major version bump requiring a manual migration. Watchtower sees one bit of information: different. It applies the same action to all of them.

It cannot know whether the new thing works. Watchtower’s success condition is that the container started. A container that starts and then fails its own health check, or starts and serves 500s, or starts and silently corrupts data with a changed schema — all of those are successes from where Watchtower is standing. There is no verification step because there is nothing generic to verify.

It cannot undo. There is no rollback. With WATCHTOWER_CLEANUP enabled, the previous image is removed, so the rollback path is a registry lookup for a digest you now have to reconstruct. Without cleanup, the image is still there, and reverting still means finding the old ID and recreating the container by hand at 4 a.m.

It cannot take a backup first. The single most valuable thing any update procedure does is snapshot the state before touching it. Watchtower has no hook for that. It stops, removes, recreates. Whatever the new version does to your data directory on first start, it does to the only copy you have.

To that, add a fifth thing that is about the ecosystem rather than the design: the project has been quiet for some time. The last tagged release is a good way back and the issue tracker reflects it. That is a real consideration for a piece of software you are handing root-equivalent access to your host and permission to change every service you run.

Because that is what this is. Mounting the Docker socket is root on the host — anything that can talk to it can start a privileged container mounting /. You are granting that, unattended, permanently, to a process whose job is to fetch and execute code from the internet on a schedule. When it is stated that plainly, the risk stops being abstract.

The defensible Watchtower

I want to be fair, because there is a configuration of Watchtower that I think is genuinely good, and it is the one nobody uses.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

services:
  watchtower:
    image: containrrr/watchtower:1.7.1
    container_name: watchtower
    restart: unless-stopped
    environment:
      WATCHTOWER_MONITOR_ONLY: "true"
      WATCHTOWER_NOTIFICATION_REPORT: "true"
      WATCHTOWER_SCHEDULE: "0 0 9 * * *"
      WATCHTOWER_NOTIFICATION_URL: "ntfy://ntfy.example.com/updates?title=Container updates"
      WATCHTOWER_NO_STARTUP_MESSAGE: "true"
      WATCHTOWER_CLEANUP: "false"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro

WATCHTOWER_MONITOR_ONLY: "true" is the whole trick. Watchtower now checks the registry, notices what has moved, tells you about it, and changes nothing at all. It has become an inventory tool. At 09:00, in the morning, a notification arrives listing what has updates. You read it over coffee, with a laptop, awake.

This costs you the fantasy of never doing maintenance and buys you the entire safety story. Every failure mode above required Watchtower to act. In monitor-only mode it acts on nothing.

The other defensible variant is label opt-in for a genuinely safe subset:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

services:
  watchtower:
    image: containrrr/watchtower:1.7.1
    environment:
      WATCHTOWER_LABEL_ENABLE: "true"
      WATCHTOWER_CLEANUP: "false"
      WATCHTOWER_SCHEDULE: "0 0 4 * * *"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro

  # opted in: stateless, no persistent data, trivially replaceable
  whoami:
    image: traefik/whoami:v1.10
    labels:
      com.centurylinklabs.watchtower.enable: "true"

Default off, opt in per container, and only opt in things that hold no state and that you would not mind restarting spontaneously. My list is short: a couple of stateless front-ends and nothing else. Anything with a data directory is excluded by policy, permanently, and WATCHTOWER_CLEANUP stays off so I keep a rollback image.

Pin the tag, and the problem shrinks

Most of the danger comes from the tag rather than from the automation, and this is the part that generalises.

postgres:latest is an instruction to run whatever the maintainers most recently pushed. postgres:16 is an instruction to run the current 16.x, which will get patch and minor updates and will never spontaneously become 17. postgres:16.8 is an instruction to run exactly that. postgres@sha256:abc... is an instruction to run exactly those bytes.

Moving from latest to a pinned major eliminates the entire category of failure I opened this post with. PostgreSQL will never surprise-upgrade you if you have told Docker which major you want. This is free, it takes an afternoon across a whole compose directory, and it is the highest-value change available. I have written about this before as part of container image housekeeping and my opinion has only hardened since.

Pinning to a full digest is the paranoid tier and worth it for anything security-relevant, because it means a compromised registry account cannot repoint a tag at something malicious that you then pull. Verify with docker image inspect and consider image signature verification on top for anything security-relevant, which closes a genuine supply-chain hole.

Digest pinning does mean you get zero updates until you act, which is its own risk. Which is why the pinning has to come with a process.

The strategy I actually run

Compose files live in git. That single sentence is doing most of the work here.

Once your infrastructure is declared in files under version control, an update becomes a diff, and a diff can be reviewed, discussed with yourself, and reverted with a command you already know. Renovate watches those files, sees that postgres:16.8 could be 16.9, and opens a pull request. The PR body contains the release notes. I read them. A patch bump gets merged in ten seconds. A major bump sits there until I have read the upgrade guide and taken a snapshot.

The apply step is a two-liner that I run deliberately:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

#!/usr/bin/env bash
# update-stack.sh — run from the stack directory
set -euo pipefail

STACK="$(basename "$PWD")"
SNAP="/mnt/backup/pre-update/${STACK}-$(date +%Y%m%d-%H%M%S)"

echo "==> Recording current digests for rollback"
docker compose config --images | while read -r img; do
  docker image inspect "$img" --format '{{.RepoTags}} {{.Id}}'
done | tee "${SNAP}.digests"

echo "==> Snapshotting volumes"
mkdir -p "$SNAP"
docker compose down
for v in $(docker compose config --volumes); do
  tar -C "/var/lib/docker/volumes/${STACK}_${v}/_data" \
      -czf "${SNAP}/${v}.tar.gz" .
done

echo "==> Pulling and starting"
docker compose pull
docker compose up -d

echo "==> Waiting for health"
sleep 20
docker compose ps --format '{{.Name}}\t{{.Status}}'

It is not elegant. It takes the services down for the duration, which for a homelab at 10 a.m. on a Saturday is a non-event. What it does is guarantee that a bad update is a tar -xzf away from being undone, and that I know exactly which digest I was on. The two things Watchtower cannot give me, in twenty lines of bash.

Verification is the other half and it is easier than people assume. Uptime Kuma already knows whether every service is answering, so the post-update check is looking at a page I already have. When something is red, Dozzle shows me why in about fifteen seconds. The tooling for a safe update is mostly tooling you already run.

Sequencing, the problem nobody mentions

There is one more failure mode worth naming because it is invisible until it bites, and it is about order.

A stack is a graph. The database starts, then the application that migrates against it, then the reverse proxy that fronts it. depends_on with condition: service_healthy expresses that, and compose honours it.

Watchtower does not see the graph. It iterates over containers and updates the ones whose digests moved, in whatever order it finds them. Update an application container while its database is mid-restart and the application runs its migrations against a database that is not accepting connections yet, fails, restarts, tries again, and — depending on how the application handles a partial migration — either recovers or leaves a schema in a state its own migration tool refuses to touch. Watchtower has a --rolling-restart flag which helps with availability and does nothing about ordering.

The reason this is invisible is that it works fine most of the time. Restart timing is a race, and races are won more often than they are lost. You lose one perhaps every twentieth update, at 4 a.m., and by the morning the container has restart-looped eighty times and the logs that would explain it have rotated away.

Compose already solved this. Declaring the dependency and letting compose bring the stack up in order is a solved problem with a well-tested implementation. Reaching around compose to mutate containers directly discards that, silently, for every stack you own:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

services:
  db:
    image: postgres:16.8
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app -d appdb"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 30s

  app:
    image: ghcr.io/example/app:2.4.1
    depends_on:
      db:
        condition: service_healthy

If your stack has a depends_on in it, you have already told the system something that Watchtower structurally cannot read. That is reason enough to keep the update path going through compose.

Troubleshooting

A container updates and immediately restarts in a loop. Almost always a config change in the new major. Check the image’s release notes for renamed environment variables — this is the single most common breakage and it is silent, because an unrecognised env var is simply ignored. Roll back to the previous digest from your notes and read properly.

Watchtower updates a container and compose reverts it later. Expected. Watchtower recreates the container directly; your compose file still declares the tag it always did. The next docker compose up -d reconciles against the file. This drift between running state and declared state is precisely the argument for putting the files in git and making them the source of truth.

Watchtower keeps pulling the same image every cycle. Multi-arch manifest confusion, usually on ARM. Watchtower compares digests, and a manifest-list digest differs from the platform-specific image digest. Pin the platform or pin the tag.

Rate limits from Docker Hub. An anonymous pull budget disappears fast when you poll fifty containers every six hours. Log in, or mirror the images you care about locally. This is a good argument for polling once a day rather than continuously.

You updated, it broke, and the old image is gone. docker image ls -a first — it may only be untagged. Failing that, your digest notes plus docker pull image@sha256:... will bring it back if the registry still has it. Failing that, you are reading changelogs at 4 a.m., which is the situation this whole post exists to prevent.

The honest verdict

The steel-man for Watchtower is real: unpatched software is a genuine risk, and the person who never updates because updating is scary ends up running a two-year-old image with a known CVE in it. Automation that mostly works beats discipline that never happens. If your alternative to Watchtower is honestly nothing, Watchtower in its default configuration is better than nothing, and I will not pretend otherwise. Scanning your images will tell you exactly how old and how exposed your never-updated stack has become.

But the gap between “never update” and “update everything unattended with no backup” is enormous, and there is a lot of good ground in it. Monitor-only mode plus a notification costs you five minutes a week and removes every failure mode in this post. Pinned majors plus Renovate plus a snapshot script costs you an afternoon to set up and gives you release notes, review and rollback. Both of those are strictly better than the default, and neither requires you to become a different person.

Who can run Watchtower as designed: someone whose containers are all stateless, all pinned to a major, all covered by a health check, and all replaceable. That configuration exists and it is fine.

Who should turn it off tonight: everyone running a database in a container with :latest and cleanup enabled. You are one registry push away from the story at the top of this article, and the reason you have not hit it yet is scheduling luck. Left alone, that is exactly the sort of quiet debt that stays invisible until the morning it is not.