Phase 3: Reliability Engineering

Error budget calculation, tracking & burn rate alerts

Intermediate ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you're building a giant, super cool LEGO tower, and you want it to be really strong and perfect. You promise your friend it will be 99.9% amazing for the next month, meaning almost every single block will be perfectly snapped into place, holding strong. But even the best builders know that sometimes, a tiny few blocks might not click in just right, or they might be a little crooked, or maybe one even falls off temporarily. These are like "oops!" moments for your tower.

That tiny bit of "not 100% perfect" – the 0.1% that's not perfectly amazing – that's your "error budget." It's like your allowance for those little "oops!" moments. It’s the small number of wobbly or imperfect blocks you're allowed to have without breaking your big promise about the super strong tower. For example, if your tower has a thousand blocks, you might have an allowance for one or two wobbly ones in a month without letting down your friend.

So, as you keep building, you need to keep an eye on these wobbly blocks. Every time you place a block and it's not absolutely perfect, you mark it down. Maybe you have a little counter, or a special jar where you put a tiny pebble for each wobbly block. This is like "tracking" your error budget. You're continuously watching to see how many "oops!" moments are happening. Each wobbly block "uses up" a piece of your allowance.

Let's say you're building really fast, and suddenly you notice your "wobbly block" jar is filling up much quicker than you expected. You might even have a little alarm set to ding if you use up half your allowance in just a day! That's like a "burn rate alert." It's a warning signal telling you, "Hey! You're making too many wobbly blocks too fast, and you're going to run out of your allowance before the month is over!" When that alarm dings, it means you need to stop and figure out why. Are you rushing? Do you need to slow down and be extra careful? Maybe you need to go back and fix some of those wobbly sections, or even change how you're building to make sure the rest of the tower is super solid. This idea helps builders know when to pause, check their work, and make sure they can still keep their promise about that amazing tower.

An Error Budget is the allowable unreliability derived directly from your Service Level Objective (SLO). If your SLO dictates 99.9% availability over a 30-day period, then the remaining 0.1% is your budget for "bad" events. Mathematically, 0.1% of 30 days translates to 43.2 minutes of unavailability. This budget isn't solely for downtime; it applies to any "bad" event your Service Level Indicator (SLI) defines, such as failed requests, requests exceeding latency thresholds, or data integrity issues. It's a precise, quantifiable amount of acceptable failure that the service can experience without violating its commitment to users.

Tracking the error budget involves continuously monitoring your SLIs to measure actual "bad" events against this predefined budget. As errors or unacceptable performance occur, they consume a portion of your budget. For example, if your SLI measures the ratio of successful requests, every failed request chips away at the budget. SRE teams use monitoring systems (like Prometheus, Grafana, or proprietary dashboards) to aggregate these bad events over the SLO period and display the remaining budget. Visualizing this as a "burn down" chart helps teams understand how much headroom they have left and predict when they might breach the SLO if the current trend continues.

While knowing the remaining budget is crucial, understanding the "burn rate" – the speed at which the budget is being consumed – is key for proactive incident response. A high burn rate indicates that the service is experiencing problems significantly faster than anticipated, potentially leading to an SLO breach well before the end of the period. Burn rate alerts are configured to notify teams when a substantial portion of the budget is consumed over a short timeframe (e.g., 5% of the total budget used in 1 hour, or 20% in 6 hours). These alerts are essential for enabling SREs to investigate and mitigate issues before a catastrophic failure or an official SLO violation occurs, ensuring reliability remains within acceptable bounds.

Key Takeaways

  • Error budget is the inverse of your SLO, representing allowable unreliability.
  • SLIs measure bad events that consume the error budget.
  • Tracking visualizes the remaining budget, often as a 'burn down' chart.
  • Burn rate is the speed at which the budget is being consumed.
  • Burn rate alerts enable proactive incident response to prevent SLO breaches.

Code Example

yaml
groups:
- name: service-slo-alerts
  rules:
  - alert: HighErrorRateBudgetBurn
    expr: |
      sum(rate(http_requests_total{job="my-service",status!~"2xx"}[5m]))
      / 
      sum(rate(http_requests_total{job="my-service"}[5m]))
      > (0.001 * 6)  # 0.1% budget, burning 6x faster than allowed (i.e., 0.6% error rate)
    for: 5m
    labels:
      severity: page
      slo_budget: "99.9%"
    annotations:
      summary: "{{ $labels.job }} is burning through its error budget quickly"
      description: "The service {{ $labels.job }} is experiencing a sustained error rate that could breach its 99.9% SLO within hours. Current error rate is {{ $value | humanizePercentage }}"

How this code works

This code defines a Prometheus alert named HighErrorRateBudgetBurn to identify when a service is consuming its error budget too quickly, signaling a potential breach of its 99.9% Service Level Objective (SLO). The alert aims to notify engineers proactively before the budget is completely exhausted. The central part is the expr field, which calculates the service's current error rate. It achieves this by taking the sum(rate(...)) of all non-2xx HTTP requests (errors) from my-service over the last five minutes and dividing that by the sum(rate(...)) of all requests from the same service in the same period.

The calculated error rate is then compared to a threshold: > (0.001 * 6). This is a crucial, subtle part. The 0.001 represents the allowed 0.1% error budget for a 99.9% SLO. Multiplying it by 6 means the alert triggers if the current error rate is six times higher than the long-term allowed average, indicating a rapid "burn rate" of the budget. The for: 5m ensures the elevated error rate is sustained for five minutes, preventing transient spikes from triggering the severity: page alert. Finally, labels like slo_budget: "99.9%" provide context, and annotations offer a clear summary and description for on-call engineers, including the actual {{ $value | humanizePercentage }} error rate.

Related Articles

Hand-picked reads to go deeper on slos, slis & error budgets.

Back to Site Reliability Engineer Roadmap