Monitoring tools watch systems and raise alerts when something crosses a threshold. When thresholds are set too low, engineers receive hundreds of alerts a day and start ignoring them, including the important ones. This is alert fatigue, and it is dangerous: the alert that predicts an outage gets lost among alerts that mean nothing.
Tuning alerts sensibly
- Alert on symptoms users feel, such as failed requests or slow responses, before alerting on every internal metric.
- Set thresholds from observed normal behaviour, not from round numbers that look tidy.
- Give every alert a runbook link: the first three steps to check, so night responders are not guessing.
- Review ignored alerts weekly and either fix the cause, raise the threshold, or delete the alert.
A quiet monitoring system is not a broken one. It is a tuned one, where every alert that fires deserves attention.