Phase 3: Reliability Engineering

Error budget policies: when to freeze deploys vs push features

Intermediate ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you're building a super cool castle with LEGO blocks. You want it to be amazing, with towers, drawbridges, and secret passages! But you also want it to be strong and not fall apart every time you touch it. Sometimes, when you're adding new, exciting parts, a wall might get a bit wobbly, or a tower leans a little. If you keep adding more and more new features – like a giant flag or a catapult – without checking if the basic structure is solid, the whole castle might collapse! That wouldn't be very fun, right? This is a bit like how the grown-ups who build big computer programs think about their own creations.

So, to make sure your LEGO castle stays standing while you're making it awesome, you could have an "Unwobbly Budget" – a special allowance for how wobbly your castle can get before you need to stop adding new stuff. If your castle is super solid, your Unwobbly Budget is full. That means you have lots of room to try out new, cool, maybe slightly risky ideas, like building a super tall spire or a tricky suspension bridge. You can add lots of new features quickly! But if you add a new piece and suddenly a wall starts to wobble a lot, you've used up some of your Unwobbly Budget. If it gets too wobbly, and you've spent almost all your budget, that's a signal.

When your castle’s "Unwobbly Budget" gets really low, or even runs out, you have a rule: stop adding new exciting features. Instead, everyone building the castle must focus on making the wobbly parts strong again. Maybe you add some reinforcing bricks, or re-stack a leaning tower. You "freeze" new features until the castle is sturdy and reliable once more. Only when the castle is strong and stable again, and your Unwobbly Budget is mostly refilled, can you go back to adding those cool new towers and secret passages. It’s all about making sure the fun new stuff doesn't accidentally break the basic, important parts of your creation.

This means that when you're building your own games or apps someday, you’ll learn that it’s important to have a plan for when to push new ideas forward and when to pause and fix things. Having a good "Unwobbly Budget" policy helps you balance making cool, new things with making sure your creations always work reliably and are fun for everyone to use, not just when they first launch!

Error budget policies are a critical component of Site Reliability Engineering that directly link your service's reliability performance to development priorities. An error budget is the maximum allowable unreliability (e.g., downtime, latency, errors) over a defined period, derived directly from your Service Level Objectives (SLOs). When a service is performing well and meeting its SLOs, its error budget is healthy, signaling that there's room to innovate, launch new features, and take calculated risks. Conversely, when the service experiences incidents or degradation, it "spends" its error budget. These policies define the specific actions taken as this budget depletes, ensuring a proactive approach to maintaining reliability.

The core dilemma these policies address is: when do we prioritize stability over new features? When the error budget is abundant, teams are encouraged to push features, perform A/B tests, and iterate quickly. However, when the budget approaches exhaustion or is completely spent, the policy triggers a shift. "Freezing deploys" is a common action, meaning that non-critical, feature-related deployments are paused or blocked. This allows engineering teams to dedicate their full attention to reliability work, such as fixing root causes of recent incidents, addressing technical debt, refactoring problematic code, or improving monitoring and alerting. The goal is to stop further potential degradation and restore the service's health, replenishing the error budget.

It's rarely an instant, hard stop. Effective error budget policies often incorporate tiered thresholds and actions. For instance, a warning might be issued when 75% of the budget is spent, leading to increased scrutiny on new deployments. A "soft freeze" might occur at 90%, requiring manager approval for any non-critical change. A "hard freeze" at 100% might block all feature deploys and mandate a reliability-focused sprint until the budget recovers. This dynamic incentivizes development teams to build reliable systems from the outset, as their ability to ship new features is directly tied to the service's reliability. It ensures a healthy tension between innovation and operational stability, aligning SRE and development goals.

Key Takeaways

  • Error budget policies link service reliability (SLO performance) directly to development activities.
  • A healthy error budget allows teams to push new features; a depleted budget triggers reliability-focused work.
  • "Freezing deploys" is a common policy when the budget is spent, halting non-critical changes to prevent further unreliability.
  • Policies often use tiered thresholds (e.g., warnings, soft freeze, hard freeze) to escalate actions as the budget depletes.
  • These policies incentivize proactive reliability engineering and align SRE and development teams on shared goals.

Code Example

yaml
errorBudgetPolicy:
  service: "api-gateway-service"
  timeWindow: "28d" # Rolling 28-day window
  budgetThresholds:
    - name: "warning-level"
      percentageSpent: 75
      action: "notify-oncall-and-leads"
      message: "Budget approaching limit, review open incidents."
    - name: "soft-freeze-level"
      percentageSpent: 90
      action: "halt-non-critical-deploys"
      message: "Non-critical deploys blocked. Focus on reliability fixes."
    - name: "hard-freeze-level"
      percentageSpent: 100
      action: "halt-all-feature-deploys-and-request-remediation-plan"
      message: "Budget exhausted. All feature deploys frozen. Critical reliability sprint required."

How this code works

This code defines an error budget policy for the specific service called "api-gateway-service". Its job is to set up automated rules for how a team should react as their error budget gets spent. The timeWindow specifies that this policy applies to budget consumption over a rolling 28-day period. The core of the policy is found in budgetThresholds, which lists different levels of budget spending and the actions to take at each point.

Each item within budgetThresholds represents a particular severity level. When the percentageSpent of the error budget reaches that amount, a specific action is triggered, accompanied by a descriptive message. For instance, at 75% spent, the policy triggers a warning-level notify-oncall-and-leads action. Reaching 90% activates a soft-freeze-level, which might halt-non-critical-deploys. Finally, exhausting the budget at 100% triggers a hard-freeze-level to halt-all-feature-deploys-and-request-remediation-plan. A subtle but critical detail is that these thresholds are generally evaluated from least to most severe, with the most severe breached threshold dictating the active policy. This ensures that if the budget hits 100%, the system correctly applies the hard freeze, rather than just adding to prior notifications.

Related Articles

Hand-picked reads to go deeper on slos, slis & error budgets.

Back to Site Reliability Engineer Roadmap