Alerting Systems

InfrastructureSenior

The Question

How do you design an effective alerting system?

What a Strong Answer Covers

• Thresholds
• escalation
• alert fatigue.

Senior-Level Answer

An effective alerting system maximises signal and minimises noise—every alert that fires should require a human response, and every condition requiring a human response should fire an alert. Achieving this balance is harder than it appears.

**Alert on symptoms, not causes**: Alert on user-visible impact (error rate, latency, availability) rather than internal resource metrics (CPU, disk usage). A CPU alert at 80% may never affect users; a 2% error rate always does. This maps alerts directly to SLOs and reduces false positives.

**SLO-based burn rate alerting**: Google's SRE model defines alerts in terms of error budget consumption. A fast burn rate alert fires when the error budget is being consumed at a rate that will exhaust it within 1 hour (high severity, page on-call). A slow burn rate alert fires when the budget will exhaust in 3 days (lower severity, ticket). This approach captures both sudden outages and slow degradations that symptom-threshold alerts miss.

**Severity tiers**: Define clear severity levels with expected response times. P1: system down, data loss risk—wake someone up. P2: degraded performance affecting users—respond within 15 minutes. P3: anomaly worth investigating—handle during business hours. Consistency in severity definitions prevents alert fatigue and sets clear on-call expectations.

**Alert routing and escalation**: Route alerts to the right team based on service ownership (PagerDuty, OpsGenie). Define escalation paths: if the primary on-call doesn't acknowledge within N minutes, escalate to the secondary. For P1s, auto-escalate to the incident commander.

**Reducing alert fatigue**: Deduplicate related alerts—a single infrastructure failure should not generate 50 application alerts. Use alert grouping and inhibition rules (Alertmanager). Set alert resolution auto-close to avoid stale firing alerts cluttering dashboards.

**Runbooks**: Every alert should link to a runbook with: what this alert means, probable causes, diagnostic steps, and remediation actions. Alerts without runbooks transfer cognitive load to the on-call engineer at 3am.

**Alert hygiene reviews**: Regularly audit firing alerts. If an alert fires frequently but requires no action, either fix the underlying issue or raise the threshold. Alerts that don't drive action are noise that erodes trust in the system.

What Separates a 2/3 from a 3/3

Common Mistakes

• Alerting on resource thresholds (CPU, memory) instead of user-visible symptoms.
• Not defining severity tiers, leading to all alerts being treated as equal urgency.
• Creating alerts without runbooks, leaving on-call engineers without context at 3am.
• Not reviewing and pruning alerts that fire frequently but require no action.

Follow-Up Questions

• What is an error budget and how does it drive burn rate alerting? — Error budget = (1 - SLO) × time window. Burn rate = actual error rate / SLO error rate. Burn rate > 1 means budget will exhaust before the window ends.
• How do you prevent a single infrastructure failure from generating 50 application alerts? — Alert grouping and inhibition rules in Alertmanager (or equivalent): if a host-down alert fires, inhibit all application alerts from that host.
• What should a good alert runbook contain? — Alert description, what it means for users, probable causes ranked by likelihood, diagnostic commands, and step-by-step remediation with rollback instructions.
• How would you handle an alert that fires every week but is always a false positive? — Either fix the underlying instability, adjust the threshold/window to reduce false positives, or deprecate the alert if it provides no signal. Document the decision.

Related Questions

• K8s — Pod vs Deployment
• K8s — Liveness vs Readiness
• K8s — ConfigMap vs Secret
• Helm Chart
• Docker — Container vs VM