Alerting That Doesn't Cry Wolf: Tuning Alertmanager

The most dangerous state a monitoring system can reach is the one where you ignore it. It doesn’t announce itself. It creeps in one over-eager alert at a time, until the notification that pops up at 2am gets swiped away on reflex because the last forty were noise, and one of those forty was your array degrading. An alert you have trained yourself to dismiss is worse than useless, because it cost you the effort of building it and it bought you a false sense of coverage.

Prometheus Alertmanager is the piece that decides which alerts become notifications, how they’re grouped, where they go, and when they’re suppressed. Getting Prometheus to detect a condition is the easy half. Getting Alertmanager to tell you about it in a way you’ll still respect after six months is the half that separates monitoring that works from monitoring that’s theatre. This is about the second half.

Why fatigue is the real failure mode

Start with the goal, because it dictates every setting that follows. The purpose of an alert is to get a human to do something. It follows that an alert which does not demand action should not exist, and an alert which fires when nothing is wrong is actively destroying the value of the ones that matter, by lowering your baseline trust in the whole system.

So the design rule is brutal and simple: every alert must be actionable, and must correspond to a real, sustained problem. If a notification arrives and the correct response is “yeah, that clears itself”, the alert is miscalibrated and needs fixing or deleting. If it fires on a momentary blip that was gone before you read it, it needs a duration threshold. The flood of true-but-pointless alerts is the real danger, because it teaches you to stop looking, and the alert you’ve trained yourself to ignore is the one that slips past when it counts. This is the same discipline that keeps Uptime Kuma’s notifications meaningful — the loud channel stays loud only if it stays rare.

Tuning happens in two places, and it helps to keep them straight. Prometheus alerting rules decide whether and when a condition counts as an alert. Alertmanager decides what happens to alerts once they’re firing — grouping, routing, suppression, timing. You need both tuned; they fix different problems.

The for duration: the single highest-value setting

Most homelab alert noise comes from conditions that are true for a moment and then aren’t. CPU hits 100% for three seconds during a build. A disk check momentarily times out. A service restarts and is unreachable for ten seconds. Alert on the instantaneous condition and you’ll be pelted with notifications for things that were fine by the time you looked.

The fix lives in the Prometheus rule, in the for clause, which requires a condition to hold continuously for a duration before the alert actually fires:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

groups:
  - name: host
    rules:
      - alert: HostHighCPU
        expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High CPU on {{ $labels.instance }}"
          description: "CPU above 90% for 10 minutes."

      - alert: HostDiskWillFillSoon
        expr: predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 24*3600) < 0
        for: 30m
        labels:
          severity: critical
        annotations:
          summary: "Disk on {{ $labels.instance }} will fill within 24h"

for: 10m on the CPU alert means a ten-minute build spike never pages you, because the condition has to persist for ten continuous minutes — which is no longer a spike, it’s a problem. The disk alert is worth a longer look: instead of alerting at a fixed percentage, predict_linear extrapolates the recent trend and fires only if the disk is on track to actually fill within a day. That’s the difference between a symptom-based alert (“disk is 85% full”, which might be its permanent healthy state) and an actionable one (“disk is filling and will run out”). Alert on the trajectory that requires action, and the 85%-forever disk never bothers you.

Grouping: one incident, one notification

When something big breaks, it breaks widely. A switch dies and thirty hosts go unreachable at once. Without grouping, that’s thirty separate notifications hitting your phone in the same second — a storm that’s harder to read than a single well-formed message and that buries the signal in its own volume.

Alertmanager’s group_by collapses related alerts into one notification. Group by the labels that define an incident, and thirty “host down” alerts arrive as one grouped notification listing thirty hosts:

1
2
3
4
5
6

route:
  group_by: ['alertname', 'cluster']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: default

The three timings do specific jobs. group_wait holds a new group briefly (30s) before sending, so alerts that fire together get bundled rather than trickling out one at a time. group_interval (5m) is how long to wait before sending an update to a group that’s already been notified — new alerts joining an existing incident. repeat_interval (4h) is how often to re-send a still-firing alert you haven’t acted on, the gentle nag that stops a real problem being forgotten without machine-gunning you every minute. Set repeat_interval too short and you’ve reinvented the noise you were trying to escape.

Routing by severity: not everything deserves the same channel

The label severity: warning versus severity: critical on those rules exists so Alertmanager can treat them differently, and it should. A critical alert — a disk about to fill, a service down that matters — earns a loud push to your phone. A warning — CPU elevated, a cert expiring in three weeks — belongs somewhere quieter you check on your own schedule.

The routing tree matches on labels and sends each class where it belongs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

route:
  receiver: default
  group_by: ['alertname']
  routes:
    - matchers:
        - severity = "critical"
      receiver: phone-push
      group_wait: 10s
      repeat_interval: 1h
    - matchers:
        - severity = "warning"
      receiver: quiet-log
      repeat_interval: 12h

receivers:
  - name: phone-push
    webhook_configs:
      - url: 'http://ntfy.mylab.local/alerts-critical'
  - name: quiet-log
    webhook_configs:
      - url: 'http://ntfy.mylab.local/alerts-warning'
  - name: default
    webhook_configs:
      - url: 'http://ntfy.mylab.local/alerts'

The critical branch pushes fast (group_wait: 10s) and nags hourly; the warning branch goes to a channel you read when you feel like it and repeats only twice a day. This tiering is the mechanism that keeps the critical channel trustworthy — because it’s rare, a buzz from it means something, and you’ll actually get up.

Annotations: make the alert tell you what to do

An actionable alert should carry the action, and Prometheus rules have an annotations block for exactly that. Whatever you put there rides along into the notification, which is where you write the thing your 2am self will thank you for — a one-line summary of what’s wrong, the specific instance affected, and a pointer to what to check or a link to a runbook. That turns a bare HostDiskWillFillSoon into a message you can act on without first working out what it even means.

1
2
3
4

        annotations:
          summary: "Disk on {{ $labels.instance }} filling — full in ~24h"
          description: "Trend predicts the root filesystem runs out within a day. Check recent large writes; prune old logs or snapshots."
          runbook: "https://wiki.mylab.local/runbooks/disk-fill"

The {{ $labels.instance }} templating fills in the specifics per firing alert, so the notification names the actual box rather than a generic class of problem. Writing the runbook pointer once, at the moment you create the alert and still remember what it’s for, saves the panicked reconstruction later when the thing actually fires and you’ve forgotten the details.

Inhibition: silence the symptoms when you know the cause

Here’s a subtle one that pays off enormously. When a host goes completely down, every service on that host also alerts — the web app is down, the database is down, the cache is down — but you don’t need five alerts, because there’s one cause and you already know it. Inhibition suppresses the downstream alerts when an upstream alert is firing:

1
2
3
4
5
6

inhibit_rules:
  - source_matchers:
      - alertname = "HostDown"
    target_matchers:
      - severity = "warning"
    equal: ['instance']

This says: when HostDown is firing for an instance, suppress any warning alert for the same instance. The host being down is the story; the twelve services that are down because the host is down are noise that would bury it. Inhibition is how you make Alertmanager tell you the root cause and stay quiet about its consequences.

Silences: the planned-maintenance mute button

You are about to reboot a box for kernel updates. You know it’ll go unreachable for two minutes and trip every alert watching it. Firing those alerts, and the ones that inhibit or group off them, is pure noise because you caused it on purpose. A silence mutes matching alerts for a set window.

Silences are created in the Alertmanager UI or via amtool, matched by label, with an expiry so you can’t forget you set one:

1
2
3
4

amtool silence add instance="mylab-01:9100" \
  --duration="1h" \
  --comment="kernel update + reboot" \
  --author="smarc"

The --duration matters. A silence with no expiry is how you end up three weeks later wondering why an actually-broken host never alerted — you muted it for a reboot and never un-muted it. Always time-box silences to a little longer than the work will take, and let them expire on their own.

Troubleshooting

An alert never fires even though the condition is clearly true. Check the rule in the Prometheus UI under Alerts — it shows whether the expression is returning data and whether the alert is pending (condition met, for duration not yet elapsed) or firing. A rule stuck pending forever means your for is longer than the condition ever persists. A rule showing no data means the expression itself is wrong or the metric doesn’t exist.

Prometheus shows an alert firing but no notification arrives. The handoff to Alertmanager or the routing is the problem. Confirm Prometheus’s alerting config points at the Alertmanager address and that they can talk. Then check Alertmanager’s own UI — if the alert appears there, it reached Alertmanager and the issue is routing or the receiver (a webhook it can’t reach); if it doesn’t appear, the alert never made it out of Prometheus.

A notification storm despite grouping. Your group_by labels are too granular — if you group by a label that’s unique per alert, every alert is its own group and grouping does nothing. Group by the labels that define the incident (alertname, and a location/cluster label) rather than by instance-specific ones.

A silenced alert still fires. The silence matchers don’t match the alert’s labels exactly. Silences match on label values precisely; a typo or a missing label means no match. Copy the labels from the firing alert in the UI rather than typing them.

Inhibition isn’t suppressing anything. The equal labels must be present and identical on both the source and target alerts. If HostDown carries instance="mylab-01:9100" but the service alert carries instance="mylab-01:8080", they don’t match on instance and inhibition won’t fire. Align the label you inhibit on.

The verdict

Alertmanager is worth learning properly, and most homelab pain with it is self-inflicted through under-tuning rather than any fault of the tool. The instinct to alert on everything is the thing to resist; a small set of alerts you trust completely beats a large set you’ve learned to ignore, every time. Spend the effort on the for durations and the predict_linear-style trajectory alerts so nothing fires on a blip, route by severity so only real problems reach your phone, and use inhibition and silences to keep incidents legible. Do that, and the notification that arrives at 2am is one you get up for — which was the entire point. Pair it with a log store you can search when the alert lands and heartbeat checks for the jobs that fail silently, and you’ve built alerting that respects your attention instead of eroding it.