Phase 5: Monitoring, Observability & Reliability

Steady-State Hypotheses & Blast Radius

Advanced ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you've built an amazing LEGO castle – towers, walls, drawbridge, everything! Before you try to make it even better, or test how strong it is, you need to know what "perfectly working" looks like. Is the drawbridge always able to go up and down? Does the main tower always stand tall without wobbling? This idea of knowing exactly what "normal and healthy" looks like for your castle is called a Steady-State Hypothesis. It's like saying, "My guess is that even if I remove one small brick from the wall, the main tower will still stand perfectly straight, and the drawbridge will still open and close smoothly for at least five minutes." You're making a clear, specific prediction about what should keep working, no matter what little tests you throw at it.

This "Steady-State Hypothesis" isn't just "the castle is generally okay." It's super specific! You might say, "The red flag on the tallest tower will stay upright, and the two knight minifigures can always walk from the gate to the throne room in under 10 seconds without bumping into anything." You write down these important things that must always be true when your castle is working right. This way, if you make a change or try to break something to test it, you instantly know if you've actually caused a problem, or if everything is still holding up just fine according to your rules.

Now, let's say you want to test how strong a particular wall of your LEGO castle is. You wouldn't just knock over the entire castle, would you? That would be a huge mess, and you'd have to rebuild everything! That potential for a big mess – how many parts of your castle could get damaged or destroyed if your test goes wrong – is called the Blast Radius. You want to make your blast radius as small as possible. Instead of demolishing the whole castle, you'd carefully remove just one specific brick from a small section of the wall to see what happens. If that tiny change causes the wall to collapse, only that small part is affected, not the whole magnificent build.

So, when engineers are building huge, complex systems, just like your LEGO castle, they use these ideas to test them safely. They first define exactly what "healthy and normal" means (their Steady-State Hypothesis), setting clear rules for how things should behave. Then, when they deliberately try to break something to find weaknesses, they make sure to limit the potential damage (the Blast Radius) by starting with tiny, controlled tests. This means they can learn how to make their systems super strong and reliable, just like you'd learn to build an un-knock-down-able LEGO castle, without ever making a gigantic mess for everyone.

In Chaos Engineering, a Steady-State Hypothesis is your bedrock. It's a precisely defined, measurable assertion about your system's normal behavior that you expect to remain true even when chaos is injected. Before you can intentionally break anything, you must understand what "working correctly" looks like. This isn't just "the server is up"; it's specific, user-centric metrics like P95 API latency remaining under 100ms, database connection pool utilization staying below 80%, or transaction success rates maintaining 99.9%. Your hypothesis might be: "Even if a single database replica fails, the P95 latency for user login will not exceed 150ms and error rates will remain below 0.1% for the next 5 minutes." This baseline gives you a clear pass/fail criterion for your experiments.

The Blast Radius defines the potential impact and scope of your chaos experiment. It's how many users, services, or components might be affected if the experiment goes sideways and causes a genuine outage. Minimizing the blast radius is paramount for running experiments safely, especially in production. You want to learn without causing widespread disruption. This means starting small: perhaps targeting a single pod in a non-critical service, a specific availability zone during off-peak hours, or a small percentage of canary traffic. Gradually expanding the blast radius as you gain confidence and validate your hypotheses is key to safely maturing your chaos program.

These two concepts are intrinsically linked. You define a steady-state hypothesis, then design an experiment with the smallest possible blast radius to test it. You inject a fault (e.g., CPU hog, network latency, pod kill) within that limited scope, and meticulously monitor your steady-state metrics. If your hypothesis holds true within the blast radius, you've validated a resilience pattern. If the steady state is violated, you've identified a weakness, learned something valuable, and can halt the experiment immediately thanks to your contained blast radius. The iterative process of testing, learning, fixing, and retesting with carefully controlled blast radii against clear steady-state hypotheses is the core of effective Chaos Engineering.

Key Takeaways

  • Steady-state hypothesis defines your system's "normal", measurable behavior.
  • Blast radius limits the potential impact of a chaos experiment to prevent widespread outages.
  • Always establish a clear steady-state baseline before injecting chaos.
  • Start with the smallest possible blast radius and expand incrementally.
  • The goal is to safely validate or invalidate hypotheses and learn about system weaknesses.

Code Example

yaml
# Example: LitmusChaos experiment definition snippet
# demonstrating blast radius control.
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: my-app-small-blast-experiment
  namespace: my-app-namespace
spec:
  # ... other ChaosEngine setup ...
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            # Control the Blast Radius:
            # Target *one specific pod* by name for initial, highly-controlled tests.
            - name: TARGET_PODS
              value: "my-service-prod-web-abcd1" 
            # Or, target a limited *number* of pods matching a selector.
            # Start with 1 to keep the blast radius minimal.
            - name: NUMBER_OF_PODS
              value: "1" 
            # Define the target application using its label
            - name: APP_LABEL
              value: "app=my-web-frontend"

How this code works

This ChaosEngine configuration snippet defines a LitmusChaos experiment designed to perform a pod-delete operation, which means it will intentionally terminate pods. The primary goal of this particular setup is to demonstrate precise blast radius control, ensuring that the chaos experiment's impact is extremely limited. The experiments section specifies pod-delete as the fault injection type, and its behavior is configured through a series of env variables within the components section. These variables dictate which pods are affected and to what extent, allowing for highly targeted disruption.

For the tightest control, TARGET_PODS allows specifying a single, exact pod name (like "my-service-prod-web-abcd1"). This ensures the experiment impacts only that specific instance. Alternatively, if a specific pod name isn't provided, APP_LABEL defines a group of target pods (e.g., all pods with app=my-web-frontend). A subtle but critical point here is how NUMBER_OF_PODS then limits how many pods from that labeled group will be affected. Starting with NUMBER_OF_PODS: "1" is a crucial best practice; it ensures that even when targeting by label, only a single, representative pod is impacted, allowing for safe, iterative testing before broadening the scope.

Related Articles

Hand-picked reads to go deeper on chaos engineering.

Back to DevOps Engineer Roadmap