Alerting rules are the core of proactive monitoring, defining specific conditions based on your metrics that, when met, signify a potential problem. In a Prometheus setup, these rules are configured in YAML files (alert.rules.yml) and evaluated by the Prometheus server. An alerting rule typically consists of an EXPR (a PromQL query) and a FOR duration. For example, you might define a rule that triggers an alert if a service's request latency (http_request_duration_seconds_bucket) consistently exceeds a certain threshold for five minutes. These rules are the "what" – they tell you what problem has occurred, and Prometheus then sends these active alerts to Alertmanager.
Escalation policies, handled by Alertmanager, dictate how these fired alerts are processed and notified. They define the "who," "when," and "how" of your incident response. Alertmanager's configuration uses routes to match alerts based on labels (e.g., severity: critical, service: frontend) and send them to specific receivers (like Slack channels, PagerDuty, or email). Policies also manage alert grouping, preventing notification storms for related issues, and repeat_interval to re-notify if an alert persists. The "escalation" comes into play when an alert isn't acknowledged or resolved within a defined timeframe, allowing Alertmanager to re-route it to a broader team or a more urgent communication channel, ensuring critical issues get the necessary attention.
Together, alerting rules and escalation policies form a robust incident detection and response system. Well-crafted rules ensure you're alerted to real problems, while effective escalation policies guarantee that the right people are informed promptly and persistently, minimizing downtime and impact. The key is to create actionable alerts that provide sufficient context for troubleshooting, avoiding "alert fatigue" from excessive or non-critical notifications. Integrating these with your incident management tools further streamlines the entire process, turning raw metric data into structured, actionable incident workflows.
Key Takeaways
- Alerting rules define the conditions (the 'what') in Prometheus based on metric expressions that trigger an alert.
- Escalation policies, managed by Alertmanager, determine how alerts are grouped, routed, and re-notified (the 'who' and 'when').
- Prometheus evaluates alerting rules; Alertmanager handles notification routing and escalation logic.
- Effective policies ensure critical issues are escalated appropriately through different channels if unaddressed.
- Prioritize actionable alerts with clear context to avoid alert fatigue and improve incident response times.
Code Example
groups:
- name: system_alerts
rules:
- alert: HighCpuUsage
expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
for: 5m
labels:
severity: critical
environment: production
annotations:
summary: "High CPU usage on instance {{ $labels.instance }}"
description: "CPU utilization on {{ $labels.instance }} has been above 90% for 5 minutes. Investigate immediately."How this code works
This configuration defines an alerting rule for Prometheus that automatically detects when a server's CPU usage is consistently high. Its main job is to monitor all production instances and trigger a "critical" alert, named HighCpuUsage, if any specific server's CPU utilization exceeds a dangerous threshold for a sustained period. This helps DevOps teams quickly identify and address performance bottlenecks before they impact users.
The expr field contains the Prometheus Query Language (PromQL) expression that calculates the CPU usage. It first measures the rate of idle CPU seconds over the last 5m and then subtracts this from 100 to get the busy percentage. This calculation is applied avg by (instance) to each server individually, ensuring the alert fires for a single problematic machine rather than an average across all. The alert then triggers if this usage goes > 90. A crucial detail is the for: 5m setting; the condition must be true for five continuous minutes before the alert is actually sent, preventing fleeting spikes from generating unnecessary noise. Once triggered, labels like severity and environment categorize the alert, and annotations provide a human-readable summary and description using dynamic values like {{ $labels.instance }} to clearly identify the affected server.