Phase 4: Incident Management

Mitigation patterns: rollbacks, feature flags & traffic shedding

Intermediate ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Have you ever been making something really cool, like a giant LEGO castle or a super-fancy cake, and suddenly, something goes wrong? Maybe a critical piece of your castle falls off, or you accidentally use salt instead of sugar in your cake batter. When that happens, you don't want to spend ages trying to fix the tiny mistake while everyone is waiting. You need to fix it fast so your castle doesn't collapse or your cake isn't ruined!

That's exactly what happens when we build computer programs. Sometimes, when we add a new feature or change something, it accidentally breaks part of our program. One of the quickest ways to fix it is called a "rollback." Imagine you're baking your favorite cookies, and you've just added a new, secret ingredient to the dough. You pop the first batch in the oven, and suddenly, you realize you made a mistake – you used salt instead of sugar! Oh no! The cookies are going to taste terrible. The fastest way to fix this isn't to try and scoop the salt out of the baking cookies. It's to quickly toss out that bad batch of dough and grab the previous batch of dough that you know was perfectly fine before you added the salt. You get a working batch into the oven right away, so nobody has to eat salty cookies.

Another super clever way to fix problems fast is using something called "feature flags." Think about those same cookies. Maybe you've perfected your recipe, but you want to try adding a brand new topping, like rainbow sprinkles. But you're not sure if everyone will love the sprinkles. Instead of putting them on all the cookies right away, you set up a little "switch" for the sprinkles. You can bake the cookies without sprinkles, and then, after they're baked, you can flip the switch "on" to add sprinkles to some, or "off" to leave them plain for others. If someone tries a cookie with sprinkles and says, "Ew, I hate sprinkles!", you can just flip the switch back to "off" for everyone else instantly. You don't have to bake a whole new batch of cookies or undo everything. You just turn that specific new topping off.

So, when we build big computer programs, these ideas let us try new things without risking everything. This means you can build amazing new features, share them with some people to test, and if something goes wrong, you can quickly go back to what worked or simply turn off the problematic part without stopping the whole show. It's like having an "undo" button and a set of on/off switches for different parts of your creation, keeping everything running smoothly for everyone who uses it.

In the heat of an incident, rapid mitigation is paramount. When a service is failing, your primary goal is to restore functionality as quickly as possible, often before a full root cause analysis is complete. One of the most direct and frequently used mitigation patterns is a rollback. If a recent deployment or configuration change is suspected to be the culprit, rolling back to the previous, known-good version is often the fastest way to stabilize the system. This effectively undoes the problematic change, allowing services to recover and providing breathing room to investigate the root cause without ongoing user impact. Automated rollback mechanisms are critical here, as manual processes are slow and error-prone during high-pressure situations.

Another powerful technique for dynamic control and mitigation is the use of feature flags (also known as feature toggles). These allow you to turn specific features or code paths on or off in production without redeploying code. If a newly deployed feature begins causing issues, you can simply toggle it off using a feature flag management system. This provides immediate relief by bypassing the problematic code. Feature flags are also invaluable for phased rollouts, A/B testing, and isolating experimental features, giving you granular control over what users experience and enabling quick disablement if a feature proves unstable or buggy.

Finally, traffic shedding (or load shedding) is a critical pattern for protecting overloaded or struggling services from complete collapse. When a service or its dependencies are overwhelmed, rather than letting it crash and burn, traffic shedding intentionally rejects a portion of incoming requests. This reduces the load, allowing the remaining requests to be processed successfully and giving the service a chance to recover or scale up. This can be implemented at various layers, from load balancers and API gateways to application-level rate limiters, often by returning a 503 Service Unavailable status code. While it means some users will be temporarily denied service, it prevents a total outage for all users and buys precious time to resolve the underlying issue.

Key Takeaways

  • Rollbacks immediately revert problematic changes to restore service.
  • Feature flags provide dynamic, real-time control to enable/disable features without redeployment.
  • Traffic shedding protects overloaded services from collapse by reducing incoming load.
  • These patterns are crucial for rapid service restoration and stability during incidents.

Code Example

python
# Assume this config comes from a feature flag service
feature_flags = {
    "enable_new_dashboard": True,
    "enable_beta_reports": False
}

def render_dashboard(user_id):
    if feature_flags.get("enable_new_dashboard", False):
        print(f"Rendering new dashboard for user {user_id}")
    else:
        print(f"Rendering old dashboard for user {user_id}")

def generate_report(report_type):
    if feature_flags.get("enable_beta_reports", False):
        if report_type == "beta":
            print(f"Generating beta report: {report_type}")
        else:
            print(f"Generating standard report: {report_type}")
    else:
        print(f"Beta reports disabled. Generating standard report: {report_type}")

# Example usage
render_dashboard(123)
generate_report("beta")

How this code works

This code demonstrates how to implement "feature flags," a powerful technique for controlling software features without deploying new code. It allows engineers to turn features on or off, or expose them to specific users, which is essential for safely rolling out new functionality, performing A/B tests, or quickly rolling back problematic features during an incident.

The feature_flags dictionary acts as a central configuration, defining the current state of different features. Functions like render_dashboard(user_id) and generate_report(report_type) then check these flags to decide which behavior to execute. For example, render_dashboard uses the enable_new_dashboard flag to show either a "new" or "old" dashboard. Similarly, generate_report checks enable_beta_reports to determine if it should generate a "beta" or "standard" report. A subtle but important detail is the use of feature_flags.get("key", False). This .get() method is chosen over direct dictionary access because it safely returns False if a flag named "key" doesn't exist, preventing errors and ensuring features remain off by default unless explicitly enabled in the configuration.

Related Articles

Hand-picked reads to go deeper on incident response.

Back to Site Reliability Engineer Roadmap