The Quiet Joy of a Server That Just Stays Up
Contents
I have a little N100 box in the cupboard under the stairs that has been running since last October. Pi-hole, a reverse proxy, a couple of monitoring containers, nothing exotic. I know it has been up that long because Uptime Kuma tells me so, in a small green number that has quietly ticked over for months without anyone touching the machine. There is no dashboard at work that gives me the same feeling that number does. It is a strange thing to be proud of, and I want to work out why it matters, because “the server didn’t crash” sounds like the lowest possible bar for an achievement.
Professionally, uptime is a means to an end. Five nines exists because a payments API going dark costs someone money every second, and the number is a proxy for revenue protected. In a homelab, nothing costs anyone money in that sense. If Pi-hole falls over at 2am, your router falls back to its configured upstream resolver and nobody notices until morning, if then. So the appeal of homelab uptime is emotional rather than economic. It is something closer to craft: a machine that keeps doing its one job, unattended, for a long stretch of time is evidence that you built the thing correctly rather than merely got it working once, on a good day, while you were watching it.
Why “it works” and “it stays working” are different projects
Getting a self-hosted service running is an afternoon’s task for most of what people run at home. Docker Compose up, point a domain at it, done. Keeping it running for a year without intervention is a different kind of work, and it happens almost entirely in the parts of the setup nobody screenshots for a homelab subreddit post: the restart policy, the log rotation, the disk that isn’t allowed to fill up, the update strategy that doesn’t leave you on an EOL kernel with a known CVE, the power supply that doesn’t brown out the board every time the kettle clicks on.
Most outages I have caused myself trace back to one of a handful of habits I have had to unlearn. Treating restart: unless-stopped as sufficient supervision, when what actually matters is whether the container comes back into a healthy state or comes back looping a crash every ten seconds while docker ps still shows it as “Up”. Letting logs grow unbounded until a full disk takes down every container on the host at once, including ones that had nothing to do with the one that was chatty. Applying security updates promptly to packages and never to the kernel, because a kernel update needs a reboot and reboots are effort, so it waits, and six months later you are three kernels behind on a box with a route to the internet. None of these are exotic failure modes. They are the same three or four mistakes, repeated across every homelab I have ever helped debug, mine included.
Building for boredom
The healthiest homelab habit I have picked up is designing every service to fail loudly and recover quietly. Loud failure means a service that stops working tells you about it within minutes, not when you happen to open the app three days later and notice it has been broken the whole time. Quiet recovery means the same service, once whatever tripped it clears, comes back on its own without a 2am SSH session and a squint at journalctl.
A systemd unit with a watchdog does both at once for anything that isn’t already containerised:
| |
The watchdog line is the part people skip. If the process itself calls sd_notify(WATCHDOG=1) on a heartbeat and stops doing so because it has wedged rather than crashed cleanly, systemd notices the missed heartbeats and kills and restarts it. Without a watchdog, a hung-but-not-dead process just sits there looking alive to every naive check, including your own eyeballs on systemctl status, which will happily report “active (running)” for a process that stopped doing anything useful hours ago.
For containers, the equivalent is a real HEALTHCHECK rather than trusting that a running process means a working one:
| |
Docker will mark the container unhealthy on repeated failures, and restart: unless-stopped combined with a healthcheck actually gets you a bounce, whereas a plain restart policy on its own only reacts to the process exiting, not to it hanging while technically alive. I learned this the hard way with a media indexer that locked up on a corrupt file roughly once a month, sat there consuming zero CPU and answering zero requests, and kept a perfectly green status in every dashboard that only checked whether the container was running rather than whether it was working.
Power is the failure mode nobody plans for
Software habits get most of the attention in homelab writing, but the single outage that has cost me the most uptime over the years traced back to something purely electrical. A cheap power strip, a kettle on the same circuit, and a mini PC with a marginal power supply is a recipe for a brief brownout that the PSU rides through fine but that the attached USB SSD does not, corrupting whatever was mid-write at the time. A basic line-interactive UPS solved this more thoroughly than any amount of software resilience could, because it removes the failure mode entirely rather than recovering gracefully from it.
The part worth doing properly is the shutdown signalling, not just the battery. A UPS that keeps a server alive for twenty extra minutes during a longer outage is only useful if something on the host is watching the UPS’s charge level and initiating a clean shutdown before the battery actually dies mid-write. NUT (Network UPS Tools) does this over USB or serial for most consumer UPS units and will broadcast a shutdown command to every machine on the same UPS, not just the one it is plugged into directly, which matters the moment you have more than one box in the rack.
Monitoring is for you, not for a customer
The other habit that pays off disproportionately is a monitoring stack that watches the monitoring stack. I run Uptime Kuma checking every internal service over HTTP and TCP, with notifications routed to a phone via ntfy rather than email, because email is where alerts go to be ignored. The trick that took me too long to learn is that Uptime Kuma watching your services is worthless if the host running Uptime Kuma is the one that goes down. A second, cheap device — even a Pi tucked on a different circuit — pinging the first one closes that gap for the price of a spare board you probably already own.
Alerts also need a threshold that respects how networks actually behave. A single missed ping over Wi-Fi is noise; three missed pings in a row from a wired device is a real problem. Tune the retry count before you tune anything else, or you will train yourself to ignore every notification within a week, which defeats the entire point of loud failure. I got this wrong initially by setting a one-minute check interval with zero retries, which produced enough false alarms from ordinary Wi-Fi flakiness that I muted the notification channel within a fortnight, which is precisely the outcome the monitoring exists to prevent.
Troubleshooting the boring failures
Most of what threatens a long uptime streak is not exciting. A disk quietly filling with Docker logs is the single most common cause I have seen, and the fix is boring: set max-size and max-file in the Docker daemon’s log driver config globally, so no individual container has to remember to do it.
| |
Thermal throttling is the second one, and it is invisible until performance degrades in ways that look like a software bug. A fanless mini PC wedged into a warm cupboard will throttle its clock under sustained load long before it shuts down, so a service that “randomly” gets slow under backup jobs is worth checking with a simple temperature poll (sensors, or vcgencmd measure_temp on a Pi) before you go hunting through application logs for a bug that was never there.
Memory leaks in long-running processes are the hardest of the three, because the symptom — the OOM killer quietly ending a process and something else restarting it — can look identical to a transient network blip in a log file if you are not watching memory over time. A weekly cron job that logs free -m output somewhere you can graph it later has caught more slow leaks for me than any APM tool, mostly because it costs nothing to leave running for months and it turns a mystery restart into an obvious upward slope on a chart.
DNS is the fourth quiet killer, and specific to anyone running their own resolver: a Pi-hole or AdGuard instance that stops updating its blocklists because a cron job silently failed will keep resolving queries just fine for months, right up until the day a blocklist source changes its URL and every request for that list starts timing out, taking your actual DNS resolution down with it. Alerting on the age of the last successful blocklist update, not just on the resolver process being alive, closes that gap.
The update problem you can automate away
The kernel-update habit I mentioned earlier deserves its own fix rather than just a confession. unattended-upgrades on Debian-based distributions will apply security patches automatically, but it will not reboot into a new kernel unless you tell it to, which is the whole reason people end up stuck three kernels behind. Setting Unattended-Upgrade::Automatic-Reboot "true" alongside a fixed reboot time keeps the box current without turning every patch Tuesday into a manual chore, and pairing it with the Uptime Kuma checks from earlier means a reboot that fails to bring services back up gets flagged within minutes rather than discovered a week later.
The objection people raise here is reasonable: automatic reboots on a box you depend on sound like trading one risk for another. In practice the risk is asymmetric. An unpatched kernel with a known local-privilege-escalation CVE sitting on a machine with several containers and at least one exposed port is a bigger and more silent risk than a scheduled 4am reboot that a healthcheck would catch within thirty seconds if it went wrong. The reboot is loud and immediate; the unpatched kernel is quiet and permanent until you remember to deal with it, which for most of us is never, until something forces the issue.
Is chasing uptime worth it
For a service your household actually depends on — DNS, a password manager, photo backup — yes, without qualification. The habits above cost an afternoon once and save you from being the reason the internet stopped working during a video call. For anything experimental you are still deciding whether to keep, no. Build it scrappy, let it fall over, and only invest in the watchdogs, the UPS signalling and the healthchecks once it has earned a permanent place in the rack. Uptime as a hobby only makes sense once uptime as a requirement already exists; chasing the number for its own sake on a service nobody depends on is a fast way to burn an evening you could have spent on something that mattered more. The quiet green number is a nice side effect of doing the unglamorous work properly, not a goal in itself, and the moment it becomes the goal is usually the moment you stop noticing when the underlying service actually breaks. Self-hosting is not free, and the time you spend on these habits is real, but it is spent once, up front, rather than repeatedly at 2am for the rest of the machine’s life. If you want the fuller accounting of what that time and the electricity actually cost against what you would otherwise pay a cloud provider, that is a separate argument, but the uptime habit is worth building regardless of which way that maths falls, because a flaky homelab is worse than no homelab at all.
Related Content
