Chaos Engineering isn't just about randomly breaking things; it's a disciplined, scientific approach to building confidence in system resilience. The core principles guide this practice, transforming ad-hoc failure injection into structured experimentation. These principles emphasize proactive exploration, running experiments in production (or production-like environments), and minimizing blast radius while maximizing learning. Fundamentally, it's about shifting from reacting to outages to proactively discovering and mitigating weaknesses before they impact users, thereby increasing the reliability of complex distributed systems.
The most critical principle is defining and observing the Steady-State Hypothesis. This is a measurable output of your system that indicates normal, healthy behavior under typical load. Think of it as your baseline. Before injecting any chaos, you define what "normal" looks like – common metrics include application throughput, latency, error rates, resource utilization (CPU, memory), or even business-specific metrics like conversion rates. Your hypothesis then states that despite introducing a controlled failure (e.g., increased latency on a dependency, CPU exhaustion on a specific service), the system's overall steady-state will be maintained or quickly return to normal. Without a clear steady-state hypothesis, you can't definitively measure the impact of your experiment or the system's recovery capabilities.
Practically, you establish a measurable steady-state, formulate a hypothesis (e.g., "If we kill N instances of Service X, the p99 latency of API Y will not exceed Z milliseconds"), and then execute the experiment. Robust observability is paramount here; you need the tools and dashboards to accurately monitor your steady-state metrics in real-time before, during, and after the experiment. If the steady-state is violated, you've identified a weakness. You then fix the underlying issue, refine your hypothesis, and iterate, continually enhancing your system's resilience and your team's confidence in its behavior during adverse conditions.
Key Takeaways
- Chaos Engineering uses a scientific method to proactively test system resilience, not just randomly break things.
- The Steady-State Hypothesis defines measurable, normal system behavior (e.g., latency, throughput) that serves as your baseline.
- Experiments aim to validate that the system's steady-state is maintained or quickly restored despite injected failures.
- Robust observability (monitoring, alerting) is fundamental for defining, tracking, and verifying the steady-state.
- It's an iterative process: define steady-state, hypothesize, experiment, identify weaknesses, fix, and repeat to build confidence.
Code Example
import requests
import time
import sys
def check_steady_state(service_url: str, expected_latency_ms: int = 200, expected_status_code: int = 200) -> bool:
"""
Checks if a service's health endpoint is operating within a defined steady-state (latency, status code).
"""
try:
start_time = time.time()
response = requests.get(f"{service_url}/health", timeout=3)
end_time = time.time()
latency_ms = (end_time - start_time) * 1000
print(f" Status Code: {response.status_code}, Expected: {expected_status_code}")
print(f" Latency: {latency_ms:.2f} ms, Expected Max: {expected_latency_ms} ms")
if response.status_code != expected_status_code or latency_ms > expected_latency_ms:
return False
return True
except requests.exceptions.RequestException as e:
print(f" ERROR: Could not reach service {service_url}: {e}")
return False
# Example usage: check if a target service is in steady-state before a chaos experiment
if __name__ == "__main__":
target_service_api = "http://your-service.example.com"
if not check_steady_state(target_service_api, expected_latency_ms=100):
print("Steady-state violated before experiment. Aborting!")
sys.exit(1)
print("Steady-state verified. Proceeding with chaos experiment...")How this code works
This Python code establishes a crucial "steady state" before a chaos engineering experiment. Its primary job is to ensure a target service is operating normally, providing a baseline to compare against when injecting faults. The main check_steady_state function performs this verification, while the if __name__ == "__main__": block demonstrates how to use it to decide whether to proceed with an experiment or abort if the service isn't stable.
The check_steady_state function works by making a requests.get call to the service's /health endpoint. It precisely measures the latency_ms using time.time() and checks the response.status_code. These measured values are then compared against expected_latency_ms and expected_status_code to determine if the service is stable. A subtle but important detail is how expected_latency_ms has a default value of 200 ms in the function's definition, but the example usage explicitly overrides it to 100 ms, showing that these parameters are fully customizable for different service requirements. The try...except block gracefully handles network failures, preventing the experiment from starting if the service is unreachable.