Comment by tryauuum
23 days ago
those alertmanager descriptions feel scary. I'm stuck in the zabbix era.
what do you mean "progressive rollout of new alerts across the organization"? what kind of alerts?
23 days ago
those alertmanager descriptions feel scary. I'm stuck in the zabbix era.
what do you mean "progressive rollout of new alerts across the organization"? what kind of alerts?
Well, all kinds. Alerting is a really great way to track things that need to change, tell people about that thing along established channels, and also tell them when it's been addressed satisfactorily. Alertmanager will already be configured with credentials and network access to PagerDuty, Slack, Jira, email, etc., and you can use something like Karma to give people interfaces to the different Alertmanagers and manage silences.
If you're deploying alerts, then yeah you want a progressive rollout just like anything else, or you run the risk of alert fatigue from false positives, which is Really Bad because it undermines faith in the alerting system.
For example, say you want to start to track, per team, how many code quality issues they have, and set thresholds above which they will get alerted. The alert will make a Jira ticket - getting code quality under control can be afforded to be scheduled into a sprint. You probably need different alert thresholds for different teams, and you want to test the waters before you start having Alertmanager make real Jira issues. So, yeah, progressive rollout.