In Site Reliability Engineering, effective incident response hinges on getting the right alert to the right person at the right time. This is precisely where Alert Routing, Escalation Policies, and Notification Channels come into play. Alert routing is the initial intelligent filter: it determines who should be the first point of contact for an alert based on its origin, severity, affected service, or any other predefined criteria. Think of it as a smart dispatcher, ensuring that an alert about a database issue goes directly to the database team's on-call engineer, preventing noise for unrelated teams.
Should that initial recipient not acknowledge or resolve the alert within a specified timeframe, Escalation Policies kick in. These policies are predefined sequences of actions designed to ensure critical alerts are never missed. An escalation policy typically outlines a series of steps: who gets notified next, how long to wait before escalating, and what communication methods to use. For example, if the primary on-call doesn't respond in 15 minutes, the alert might escalate to a secondary on-call engineer; if still unaddressed after another 30 minutes, it might involve a team lead or even a broader team, ensuring accountability and continuous coverage.
Finally, Notification Channels are the actual delivery mechanisms through which these alerts and escalations reach their human targets. These can range from high-urgency methods like automated phone calls and SMS messages (for critical incidents) to less intrusive options like email, Slack/Microsoft Teams messages, or mobile app push notifications (for informational or lower-severity issues). The choice of channel is crucial for managing alert fatigue and ensuring urgent issues grab immediate attention, while non-critical updates can be reviewed at convenience. Together, these three components form the backbone of a robust and reliable on-call system, minimizing mean time to acknowledge (MTTA) and mean time to resolution (MTTR).
Key Takeaways
- Alert routing directs an alert to the correct initial team or individual based on its context.
- Escalation policies define timed sequences of notifications if an alert isn't acknowledged.
- Notification channels are the delivery methods (calls, SMS, email, chat) used to reach individuals.
- Proper configuration ensures critical alerts are always addressed, minimizing MTTA and MTTR.
- Thoughtful channel selection is vital for balancing urgency with avoiding alert fatigue.
Code Example
name: Critical Service Escalation
teams:
- primary_sre_team
escalation_rules:
- rule_index: 1
delay_in_minutes: 0
targets:
- type: schedule
id: P0C1K2L # Primary On-Call Schedule
- rule_index: 2
delay_in_minutes: 15
targets:
- type: user_group
id: S3T4U5V # Secondary SRE On-Call Group
- rule_index: 3
delay_in_minutes: 30
targets:
- type: user
id: M6N7O8P # SRE ManagerHow this code works
This YAML defines an escalation_policy for critical services, ensuring that alerts reach the right people in a structured way. It outlines a sequence of steps to follow when an incident occurs, starting with the immediate on-call team and escalating to broader teams or individuals if the alert isn't acknowledged. The policy is named Critical Service Escalation and is associated with the primary_sre_team, indicating its ownership.
The core of the policy is the escalation_rules list, which specifies the order and timing for notifications. The first rule, with a delay_in_minutes of 0, immediately targets the Primary On-Call Schedule. If the alert isn't resolved within 15 minutes, rule_index: 2 activates, notifying the Secondary SRE On-Call Group via a user_group target. Finally, if 30 minutes pass without resolution, rule_index: 3 involves the SRE Manager as a specific user target. A subtle but crucial detail is the delay_in_minutes for the first rule: a value of 0 means the alert is sent right away, without any initial waiting period, which is essential for truly critical incidents.