Phase 2: Observability

Alert design: symptom-based vs cause-based alerting

Intermediate ~4 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you have a super special garden, full of beautiful flowers and yummy vegetables. Your job is to make sure this garden always looks amazing and gives people joy. To do this, you need to know quickly when something isn't quite right. There are two main ways you can get a warning about a problem in your garden.

One way is to notice that a plant itself is looking sad. Maybe its leaves are turning yellow, or it’s drooping, or the flowers aren't opening up. This is like a "symptom alert." It tells you exactly that there's a problem right now affecting what people see and enjoy about your garden. When your garden starts to look less beautiful for visitors, that's a big sign you need to jump in and help, because the joy it gives is starting to go away.

The other way to get a warning is to check things that could cause a plant to get sick, even before it starts looking droopy. For example, maybe you notice the soil is really dry, or the sprinkler isn't spraying enough water, or a few tiny bugs are starting to munch on a leaf. These are like "cause alerts." They tell you why a plant might get sick later, or warn you about a potential problem before anyone even sees a sad-looking flower.

Now, if you're the garden helper, which alert makes you drop everything and run to the garden right away? Usually, it's when a plant is actually wilting and looking sad (the symptom alert). That means the garden isn't bringing joy anymore, and people are noticing. While checking for dry soil or bugs (the cause alerts) is super smart for preventing future problems, the most urgent alarms are the ones that tell you a real, visible problem is happening right now. So, when you're deciding how your system should tell you about trouble, think about whether it’s telling you something bad has already started to happen for its "visitors," or just warning about a possible future issue.

In Site Reliability Engineering, a critical decision in alert design is whether to focus on symptoms or causes. Symptom-based alerting focuses on the observable impact on users or the system's external behavior, directly reflecting when a Service Level Indicator (SLI) or Service Level Objective (SLO) is being violated. Think of it as "what the user experiences" – like high latency, increased error rates, or a service becoming unavailable. These alerts tell you that there's a problem affecting the system's functionality. Conversely, cause-based alerting focuses on the internal state or specific components that might lead to a problem, such as high CPU utilization on a server, a database connection pool being exhausted, or a specific dependency failing. These alerts tell you why something might be going wrong, or warn you about potential issues before they manifest as user-facing problems.

For on-call SREs, prioritizing symptom-based alerts is paramount. When your pager goes off, you want it to be because there's a confirmed impact on users or critical system functionality, not just because an internal metric looks slightly off. Symptom alerts are generally more actionable, directly indicating user pain, and are more resilient to underlying infrastructure changes. For instance, if you have an alert for high error rates, it doesn't matter if it's due to a faulty deployment, a network issue, or a database problem – the immediate concern is the user impact, and the alert guides you to investigate the underlying cause. This approach reduces alert fatigue by ensuring that pages are for genuine, user-affecting issues.

While symptom-based alerts drive your immediate response, cause-based alerts still play a vital role, primarily in diagnostics and as supplementary, lower-severity signals. Once a symptom-based alert fires, cause-based alerts (or simply reviewing cause-related metrics) become invaluable for quickly pinpointing the root cause. For example, a high latency alert might trigger your pager, and then checking metrics for high CPU usage, slow database queries, or network saturation (all cause-based indicators) helps you narrow down the problem. Furthermore, some cause-based alerts can serve as early warnings for non-critical issues, allowing proactive intervention before a symptom-based alert is triggered, but these should rarely be configured to page an on-call engineer directly unless the impact is truly inevitable and severe.

Key Takeaways

  • Symptom-based alerts focus on user impact (e.g., high latency, errors) and are preferred for on-call paging.
  • Cause-based alerts focus on internal states (e.g., high CPU, disk full) and are valuable for diagnostics and early warnings.
  • Prioritize symptom-based alerts for pages to reduce alert fatigue and ensure actionable responses to user-affecting issues.
  • Use cause-based alerts to help diagnose the root cause after a symptom alert fires, or for non-critical informational purposes.
  • An effective SRE alerting strategy combines both, with symptoms driving immediate action and causes aiding efficient resolution.

Code Example

yaml
# Symptom-based alert: High application error rate
- alert: HighApplicationErrorRate
  expr: sum(rate(http_requests_total{status="5xx"}[5m])) by (job) / sum(rate(http_requests_total[5m])) by (job) > 0.05
  for: 2m
  labels:
    severity: page
    impact: "Users are experiencing 5xx errors."
  annotations:
    summary: "High 5xx error rate detected for {{ $labels.job }}"
    description: "The 5xx error rate for service {{ $labels.job }} has been above 5% for 2 minutes. This indicates a user-impacting issue."

# Cause-based alert: High CPU utilization
- alert: HighServerCPU
  expr: (100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)) > 90
  for: 5m
  labels:
    severity: warning
    impact: "Potential performance degradation if not addressed."
  annotations:
    summary: "High CPU utilization on {{ $labels.instance }}"
    description: "CPU utilization on host {{ $labels.instance }} has been above 90% for 5 minutes. This might impact service performance."

How this code works

This YAML configuration defines two Prometheus alert rules, enabling systems to automatically detect and notify about potential problems based on specific metric thresholds. The first rule, named HighApplicationErrorRate, exemplifies a symptom-based alert. Its expr query calculates the percentage of 5xx HTTP errors over a five-minute window for each application job. If this error rate exceeds 5% (> 0.05), the alert triggers, signaling that users are actively encountering issues, as noted in its impact label. The severity: page label indicates this is a critical, user-facing problem requiring immediate attention.

The second rule, HighServerCPU, demonstrates a cause-based alert. Its expr query monitors CPU utilization on server instances, calculating usage by looking at the idle CPU rate and subtracting it from 100%. If CPU usage remains above 90% for five minutes, a severity: warning alert is issued, flagging a potential performance bottleneck before it necessarily causes user-facing symptoms. A subtle but crucial detail in both alerts is the for duration. This ensures an alert only fires if the condition persists for the specified time (e.g., 2 minutes for errors, 5 minutes for CPU), preventing transient spikes from triggering unnecessary alerts and leading to alert fatigue.

Related Articles

Hand-picked reads to go deeper on alerting & on-call tooling.

Back to Site Reliability Engineer Roadmap