Alerting That Doesn't Cry Wolf: Tuning Alertmanager

Fri, 09 Aug 2024 08:09:00 +0000

The most dangerous state a monitoring system can reach is the one where you ignore it. It doesn’t announce itself. It creeps in one over-eager alert at a time, until the notification that pops up at 2am gets swiped away on reflex because the last forty were noise, and one of those forty was your array degrading. An alert you have trained yourself to dismiss is worse than useless, because it cost you the effort of building it and it bought you a false sense of coverage.

Alertmanager - vo.rs

Alerting That Doesn't Cry Wolf: Tuning Alertmanager