Automatic Security Updates Without the 3am Reboot Surprise

There are two ways to run an unpatched box. You can patch everything by hand, fall behind the moment life gets busy, and eventually discover your services have been sitting on a two-month-old openssl for reasons you cannot reconstruct. Or you can automate the patching, wake up to a service that vanished overnight because a kernel update rebooted the machine mid-backup, and swear off automation forever. Both roads lead to the same conclusion drawn too hastily: that keeping systems current at home is more trouble than it is worth.

The trouble is real, and reboots cause almost all of it. Installing an updated library is fast, safe and reversible. Restarting the process that uses it, or rebooting the whole host to load a new kernel, is where the outage hides. So the strategy I have settled on separates those two acts completely: let the machine patch itself continuously and quietly, and keep the decision about when to reboot firmly under my control, on a schedule I choose.

Why automate patching but not rebooting

The case for automatic security updates is the same automated scanning that makes them necessary. The bots probing your internet-facing services do not read the CVE feed and then think about it — they weaponise a fix within days of it landing, sometimes hours, and spray the internet indiscriminately. A homelab reverse proxy is not too small to be found; it is found by the same mass scans that find everyone. Manual patching loses that race whenever you are on holiday, ill, or simply busy for a fortnight, which over a year is a lot of exposure. Working out which patches are urgent is a separate skill I cover in CVE triage for a homelab; this post is about applying the routine ones without thinking, so triage is reserved for the genuinely interesting cases.

The case against automatic reboots is that a reboot is the single most disruptive thing that can happen to a running service, and it should never happen at a time you did not choose. A database mid-write, a backup mid-transfer, a long-running job three hours into a four-hour run — a surprise reboot turns any of these into a bad morning. The machine has no idea what it is interrupting. You do. So the machine gets to install packages freely and gets to tell you a reboot is needed, and you get to decide the moment.

Setting up unattended-upgrades the sensible way

On Debian and Ubuntu the tool is unattended-upgrades, and its defaults are close to right but worth understanding rather than blindly accepting. The key file is /etc/apt/apt.conf.d/50unattended-upgrades. Here is the shape I run, trimmed to the lines that carry a decision:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

// /etc/apt/apt.conf.d/50unattended-upgrades

Unattended-Upgrade::Origins-Pattern {
    // Security updates only. Do NOT auto-pull feature updates —
    // those are the ones that break things.
    "origin=Debian,codename=${distro_codename}-security,label=Debian-Security";
};

// Clean up so /boot and disk don't quietly fill with old kernels.
Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";
Unattended-Upgrade::Remove-Unused-Dependencies "true";

// Reboot handling: DO detect that one is needed, do NOT perform it.
Unattended-Upgrade::Automatic-Reboot "false";

// Email me when something happened or something failed.
Unattended-Upgrade::Mail "root";
Unattended-Upgrade::MailReport "only-on-error";

// Never touch a package I've deliberately held back.
Unattended-Upgrade::Package-Blacklist {
    // "docker-ce";   // pin container engine, upgrade it deliberately
};

The single most important line is Automatic-Reboot "false". That is the difference between a patching system and a 3am roulette wheel. The second most important is restricting Origins-Pattern to the security pocket. Letting unattended-upgrades pull every available update, including feature releases, is how a routine overnight run turns into an unplanned major-version bump of some daemon that then refuses to start. Security-only keeps the automated stream boring, and boring is exactly what you want running unsupervised.

The actual schedule lives in /etc/apt/apt.conf.d/20auto-upgrades, which enables the daily systemd timers:

1
2
3

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
APT::Periodic::AutocleanInterval "7";

On Fedora, RHEL and their relatives the equivalent is dnf-automatic. Its config in /etc/dnf/automatic.conf has a matching philosophy: set upgrade_type = security and apply_updates = yes to install patches, while leaving the box to signal — rather than force — any reboot. The tool differs, the discipline is identical.

Restarting services without a full reboot

A great many security patches never need a reboot at all. If openssl is updated, you do not have to restart the whole machine — you have to restart the processes that loaded the old openssl into memory. The difference between “restart three daemons” and “reboot the host” is the difference between a two-second blip and a two-minute outage plus whatever was interrupted.

needrestart is the tool that closes this gap. After an upgrade it works out which running processes are still using deleted library versions and restarts just those services, leaving everything else untouched. In automatic mode it does this without prompting:

1
2
3
4
5

sudo apt install needrestart

# /etc/needrestart/needrestart.conf
# $nrconf{restart} = 'a';   # 'a' = automatically restart affected services
# $nrconf{kernelhints} = -1; # stay quiet about kernel; we handle reboots ourselves

With needrestart handling library-level restarts, the only thing left that genuinely requires a full reboot is a kernel update. And kernel updates are precisely the ones worth scheduling deliberately, because they are the ones that reboot the whole box.

Owning the reboot window

So the machine now patches itself, restarts affected services on its own, and drops a flag at /var/run/reboot-required when a kernel or other core change needs a full restart. The remaining job is to reboot on my terms. For a single host, a weekly maintenance window driven by a systemd timer is plenty:

1
2
3
4
5
6
7
8

# /etc/systemd/system/scheduled-reboot.service
[Unit]
Description=Reboot if a kernel/core update requires it
ConditionPathExists=/var/run/reboot-required

[Service]
Type=oneshot
ExecStart=/sbin/shutdown -r +5 "Scheduled maintenance reboot for pending updates"

1
2
3
4
5
6
7
8

# /etc/systemd/system/scheduled-reboot.timer
[Timer]
# Sunday 04:00 — after backups finish, before anyone is awake to notice.
OnCalendar=Sun *-*-* 04:00:00
Persistent=true

[Install]
WantedBy=timers.target

The ConditionPathExists line is the quiet clever bit: the service only fires if a reboot is genuinely pending, so on the many weeks nothing needs one, the timer wakes up and does nothing. You are never rebooting for the sake of it. Pick a window that sits after your backups complete, so a reboot can never orphan a backup halfway through — coordinating those two schedules is exactly the kind of thing worth writing down alongside the rest of your stack conventions, in the spirit of Docker Compose patterns that age well.

For a Kubernetes homelab the same idea has a purpose-built tool. kured (the Kubernetes Reboot Daemon) watches every node for that same reboot-required flag, and when it finds one it cordons and drains the node, reboots it, then uncordons it — one node at a time, respecting a configurable window and a lock so two nodes never go down together. That is the cluster-scale version of “patch freely, reboot on a leash”, and it means a kernel CVE across a five-node cluster gets handled overnight with no dropped workloads and no involvement from me.

The gap: automatic updates stop at the host

Here is the thing that catches people out, and it is worth saying plainly before you declare the job done. unattended-upgrades and dnf-automatic patch the host operating system. They do absolutely nothing for the software running inside your containers. If your reverse proxy, your database and your web app all run as containers — as most homelab services now do — then a patched host still runs a months-old openssl inside every one of those images, and your scanner will happily confirm it.

Container images are updated a different way: you pull a fresh base, rebuild, and redeploy. There is no in-place apt upgrade for a running container that survives a restart, because the container’s filesystem resets to the image on every recreate. So the container half of your patching needs its own automation, and it looks like a scheduled rebuild rather than a scheduled apt run. A small job that rebuilds your images from current bases on a weekly cadence and redeploys them is the container-world equivalent of the host timer above. This is one more reason to put your builds behind CI: a pipeline that rebuilds on a schedule, runs the tests, and only promotes an image that passes gives you the same hands-off patching for containers that unattended-upgrades gives you for the host.

Two things make that container patching far less painful. The first is a minimal base image, so there is less inside each container to be vulnerable in the first place — I make the case for that in distroless and multi-stage builds. The second is pinning base images to a digest and bumping them deliberately, so a rebuild is reproducible and you always know exactly which base you shipped. Between a patched host and a rebuild-on-schedule pipeline for the containers, both halves of the system stay current without either one waking you up.

Troubleshooting the failure modes

A package is held back and never upgrades. Run unattended-upgrade --dry-run --debug and read what it decided. Usually the package is caught by your blacklist, or it is a “phased” update that Ubuntu is deliberately rolling out to a percentage of machines and yours has not been selected yet. Phased rollouts are working as intended — the update will arrive within a few days. Do not force it just because it is “missing”.

Services did not come back after needrestart restarted them. A daemon that fails to restart cleanly under needrestart was probably relying on something that changed underneath it. Check journalctl -u <service> for the actual error. This is why I set MailReport to email on error: a silent restart that failed is the one that bites, and I would rather get the mail than discover it when someone tells me the site is down.

The box rebooted when I told it not to. Confirm Automatic-Reboot really is false — a distro upgrade or a config-management run can quietly reset the file to a packaged default. It is worth grepping the whole /etc/apt/apt.conf.d/ directory, because a stray override in a higher-numbered file (say 52-my-overrides) wins over the one you edited, and that precedence surprises people.

/boot filled up and apt now refuses to work. Old kernels accumulated because Remove-Unused-Kernel-Packages was not set. Clear the backlog with apt autoremove --purge, then enable that line so it stays clear. A full /boot mid-upgrade can leave a half-configured kernel, so fix the space first and run apt --fix-broken install before anything else.

Everything works but you have no idea whether it ran. Silent success is its own hazard, because you cannot tell a healthy quiet machine from a broken timer. Check systemctl list-timers to confirm the apt timers are actually scheduled, and glance at /var/log/unattended-upgrades/unattended-upgrades.log occasionally. A patching system you never verify is a patching system you are trusting on faith.

Is it worth it?

For anything you would be annoyed to lose or embarrassed to have compromised, yes, without hesitation. The setup is an afternoon and the config above barely changes across machines, so it amortises to almost nothing per host. The payoff is that the tedious, urgent, easy-to-forget half of security maintenance simply happens, continuously, whether or not you are paying attention, and the disruptive half stays scheduled on a night you chose.

The mistake to avoid is the all-or-nothing version — either refusing to automate because you fear surprise reboots, or automating so aggressively that the machine reboots itself whenever it likes. The whole value is in the seam between those two: automatic where it is safe, manual where it hurts. Get that seam right and you stop thinking about patching altogether, which for a homelab is exactly the outcome you want, because the hours you are not spending on openssl are hours you can spend on the parts of the lab that are actually fun.