Self-Hosted Dead-Man Switches for Your Cron Jobs
The backup that silently stopped running six weeks ago, and the tiny service that would have told you
Contents
The worst homelab failures are the silent ones. A service that crashes loudly gets noticed. A backup job that simply stops running makes no noise at all — no error, no alert, no broken page — and you carry on believing you’re protected right up until the day you need the backup and discover the last good one is six weeks old. The job didn’t fail. It just quietly ceased, and nothing was watching for its absence.
This is cron’s original sin, and it’s worth understanding precisely because the fix follows directly from it. A dead-man switch — the same idea as the pedal that stops a train if the driver lets go — flips monitoring on its head: instead of alerting when something goes wrong, it alerts when it stops hearing that things are right. The job’s job becomes to check in. Silence is the alarm.
Why cron fails silently
Cron’s error handling is a relic, and knowing exactly how it works tells you why it can’t be trusted to tell you about failure. Cron captures a job’s standard output and error, and if there’s any output, it emails it — to the local user, via a mail transfer agent, that on a modern minimal server frequently isn’t even installed. So the failure modes stack up like this. The job runs and errors, but produces no output, so cron emails nothing. Or it errors with output, cron tries to email, there’s no MTA, and the mail vanishes. Or — the deadliest case — the job doesn’t run at all, because cron itself was stopped, or the host was down, or someone commented out the crontab line while debugging and forgot. A job that never runs produces no output to email about. There is nothing to notice.
Every one of these is invisible to any monitoring that watches for errors, because there is no error to see. You cannot alert on a log line that was never written. This is why a backup can rot for weeks: the absence of the job leaves no trace, and absence is precisely what conventional alerting can’t detect. The only way to catch a job that stopped is to expect it to announce itself and to notice when it doesn’t — which conventional threshold alerting isn’t shaped to do, because it fires on a bad value rather than a missing check-in.
The inversion: absence as signal
A dead-man switch is a small service that expects a periodic ping. You tell it “this job checks in every day”. The job, on success, sends an HTTP request to a unique URL. The service records the check-in. If a check-in doesn’t arrive within the expected window plus a grace period, the service alerts you.
The elegance is that it monitors the thing conventional tools can’t: non-occurrence. If the job runs and succeeds, it pings, and all is quiet. If the job errors before the ping, no ping arrives and you’re told. If cron never fired the job, no ping arrives and you’re told. If the whole host is down, no ping arrives and you’re told. Every silent-failure mode collapses into one detectable event — the missing check-in — and one alert covers them all.
I touched on doing this with Uptime Kuma’s push monitors in the status-page piece, and for a couple of jobs that’s a perfectly good answer. When you have a dozen scheduled jobs across several hosts and want cron-schedule awareness, per-job grace periods and proper history, a dedicated tool earns its place. The best self-hosted one is Healthchecks, an open-source project (by Pēteris Caune) that you run yourself — the same software behind the hosted healthchecks.io, MIT-licensed and designed to be self-hosted from day one.
Standing up Healthchecks
It’s a Django application. The container needs a database (SQLite is fine to start; Postgres if you’ll lean on it) and a couple of environment variables so it knows its own URL and how to send mail.
| |
SITE_ROOT matters more than it looks: it’s the base URL Healthchecks uses to build the ping URLs it hands you and the links in its alert emails, so set it to however you actually reach the service. Bring it up, log in, and create your first check. Healthchecks gives you a unique ping URL for it, something like https://hc.mylab.local/ping/<uuid>, which is the address your job will call.
Wiring a job
The simplest integration is a single curl at the end of the job. Append it to the crontab line so it only runs if the job before it succeeded:
| |
The && is doing the real work: curl runs only if backup.sh returns success, so a failed backup withholds the ping and Healthchecks raises the alarm when the check-in doesn’t come. The curl flags matter too — -f fails on HTTP errors, -m 10 caps the whole thing at ten seconds so a hung ping can’t stall cron, and --retry 3 rides out a momentary network blip so you don’t get a false alarm because the ping itself failed to send.
For anything more than a trivial job, wrap it so Healthchecks knows whether it succeeded, when it started, and what went wrong if it failed. Healthchecks exposes /start and /fail endpoints alongside the success ping:
| |
Now a failure gives you an alert that includes what the job actually said on the way down, so the notification is diagnostic instead of just “something’s wrong”. The /start ping means Healthchecks can also catch a job that begins and then hangs forever without finishing — a case a success-only ping would miss entirely. Pipe that captured output into your log store as well and you’ve got both the alert and the searchable record.
Schedules and grace periods
The tuning that separates a useful dead-man switch from a noisy one lives in two settings per check. Healthchecks can be told a cron schedule directly — give it the same 30 2 * * * your crontab uses and it knows precisely when to expect each check-in, rather than a naive “every 24 hours”. That precision means a job scheduled for 2:30am isn’t considered late at 2:29am.
The grace period is how long after the expected time Healthchecks waits before declaring the check down. Set it to a comfortable margin over the job’s normal runtime. A backup that usually takes twenty minutes wants a grace period well clear of that, so a slightly-slow-tonight run doesn’t page you, while a genuinely-didn’t-run job still gets caught within the hour. Too tight and you get false alarms on slow nights; too loose and a dead job sits undetected for longer than you’d like. Match the grace to the job’s real variability.
Alerting integrations
A check going down is only useful if the alert reaches you, and Healthchecks integrates with the usual suspects — email out of the box, plus webhooks, ntfy, Telegram, Slack, Discord, Pushover and more. Route the important jobs (backups, above all) to a channel that actually gets your attention, the same tiering logic that keeps any alert channel trustworthy. A dead backup deserves the loud channel; a cosmetic cleanup script that skipped a night can go somewhere quieter.
The chicken-and-egg problem
There’s an obvious catch worth naming: who watches the watcher? If Healthchecks itself goes down, it stops receiving pings, but it also stops sending alerts about missing pings — so a dead monitor could look identical to a healthy quiet night. Two things address this. First, put a plain uptime check on the Healthchecks service from a different monitor (an Uptime Kuma HTTP check, say) so you’re told if the dead-man switch itself dies. Second, the hosted healthchecks.io offers a paid feature to alert if your self-hosted instance stops reporting; for a homelab, a cross-check from a second, independent monitor is the free and sufficient version. Monitoring always bottoms out in “something must watch the last watcher”, and a second small tool watching the first is where that regress sensibly stops.
Troubleshooting
A check shows down but the job definitely ran. The ping isn’t reaching Healthchecks. Run the exact curl from the job’s host by hand and watch it — a firewall, a DNS problem resolving SITE_ROOT, or TLS trouble on the reverse proxy in front of Healthchecks are the usual causes. Because the job’s curl output is discarded to /dev/null, a failing ping is invisible until you run it manually.
Constant false “late” alerts on a job that’s fine. The schedule or grace period is wrong. Confirm the cron expression in Healthchecks matches the crontab exactly, and that the timezone set on the check matches the host’s — a check expecting UTC while the host runs local time will look late by the offset. Widen the grace period if the job’s runtime varies more than you allowed for.
The job fails but you get no failure detail. You’re using a bare success ping, so a failure just withholds it and you get a generic “down” with no context. Switch to the wrapper form with /fail and --data-raw so the job’s output rides along with the alert.
Pings arrive but Healthchecks isn’t emailing. Mail configuration. The EMAIL_* variables must point at a working SMTP server with valid credentials; test by triggering a check down deliberately. If email is painful, skip it and use a webhook-based channel like ntfy instead, which sidesteps SMTP entirely.
A hung job never alerts. A success-only ping can’t detect a job that starts and never finishes, because no success ping is expected until it completes. Add the /start ping and set a check that expects completion within a window, so a job that begins and hangs trips the alarm.
The verdict
A dead-man switch is one of those tools whose value is invisible until the day it saves you, and then it’s worth more than everything else in the rack combined. Cron’s silent-failure problem is real, it’s decades old, and it’s still catching people out with rotted backups they trusted. Self-hosting Healthchecks takes an evening, wiring each job is a one-line curl or a short wrapper, and the payoff is that “the backup silently stopped six weeks ago” stops being a sentence you can ever say. It’s the natural companion to a status page and alerting you’ve tuned to be trustworthy: those watch the things that announce their failure, and this watches the things that fail by going quiet. Every homelab has jobs that fail silently. This is how you find out before it matters.
Related Content
