Phase 3: Reliability Engineering

Designing experiments: pod kills, network partitions & latency injection

Advanced ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you’ve spent weeks building the most incredible LEGO city ever. It’s huge, with tall skyscrapers, busy roads, a bustling airport, and even a tiny hospital. Everything is working perfectly, all your little LEGO people are happy, and you’re super proud. This perfect, busy city is like a computer system that’s running smoothly, with all its different parts working together to make a website or an app work for lots of people. Now, here’s a secret: even the best LEGO cities and computer systems can sometimes have problems. A building might accidentally fall over, or a road might get blocked. Instead of just hoping nothing bad ever happens, smart engineers like to pretend things will go wrong, on purpose, to see if their systems are strong enough to fix themselves! It’s like testing your LEGO city to make sure it won’t fall apart the moment a tiny LEGO car bumps into a building.

One way we do this is by carefully, and on purpose, knocking over a single building in our LEGO city – like the main post office. In the computer world, this is called a 'pod kill,' where a 'pod' is like one of those important buildings or a tiny team of LEGO builders working on a specific task. We expect that when the post office falls, your amazing ‘city manager’ (a special computer program that looks after everything) should notice right away. It should then quickly find a new spot, grab some fresh LEGO bricks, and build a brand new post office, often even better than before, without anyone even realizing the old one was gone! We watch to see if mail still gets delivered, how fast the new post office appears, and if any LEGO people get confused.

But what if the problem isn’t just one building falling? What if a whole section of your city, like the busy downtown area, suddenly can’t send messages or share roads with the airport on the other side? All the buildings are still standing, but they can’t talk to each other. This is like a 'network partition' in computer systems, where different parts of a website or app can't communicate. We'd pretend a big earthquake just opened up a giant canyon between downtown and the airport, blocking all direct roads and communication lines. We want to see if your LEGO city's systems are smart enough to find new, roundabout ways for people and supplies to travel, or if the airport can still function mostly on its own, so flights don't get canceled and things don't completely stop.

By doing these kinds of careful 'disaster drills' with our LEGO cities (or computer systems), we learn exactly how strong they are. We discover hidden weaknesses we didn’t know about and then we can make them even better and tougher. So, when you’re building your own cool apps or websites someday, you’ll know that by carefully testing how they react when things go wrong, you can make sure they’re super reliable and keep working smoothly for everyone, no matter what happens!

Designing effective Chaos Engineering experiments moves beyond just "breaking things" to systematically validating system resilience against anticipated failures. Each experiment should begin with a clear hypothesis about how your system should behave under specific fault conditions, starting from a defined steady-state. For modern cloud-native architectures, pod kills are a foundational experiment. They simulate the most common failure scenarios: container crashes, node failures, or graceful terminations during scaling events. The goal is to verify your orchestrator (like Kubernetes) correctly detects the failure, reschedules workloads, and ensures service continuity without manual intervention or user impact. Observe metrics like service availability, error rates, pod restart counts, and resource utilization during recovery to confirm your service's self-healing capabilities.

Beyond individual component failures, network issues are a pervasive and often insidious cause of outages. Network partition experiments simulate scenarios where services or entire nodes become isolated, mimicking datacenter splits or firewall misconfigurations. This tests the resilience of distributed systems to communication loss, verifying how services handle unreachable dependencies, timeouts, and fallback mechanisms. Latency injection, on the other hand, simulates network congestion, overloaded proxies, or inter-region communication slowdowns. It's crucial for understanding how your system behaves under degraded network conditions, revealing cascading timeout issues, thread pool exhaustion, or improper retry logic. These experiments validate the robustness of your circuit breakers, retry policies, and overall asynchronous communication patterns.

When designing these experiments, always consider the blast radius – start small, ideally in non-production environments, before gradually expanding. Define clear pre-conditions (e.g., all services healthy, required resource levels) and post-conditions (expected system state after the experiment). Crucially, ensure robust observability is in place to accurately measure the impact and validate or refute your hypothesis. This means monitoring application metrics, system logs, infrastructure health, and end-user experience. Always have a clear rollback plan to stop the experiment and restore normalcy if unintended consequences arise. Tools like Chaos Mesh or LitmusChaos provide frameworks to define, execute, and observe these sophisticated fault injection scenarios systematically.

Key Takeaways

  • Always start with a clear hypothesis and define the system's steady-state behavior.
  • Pod kills validate fundamental self-healing and service orchestration resilience.
  • Network partitions and latency injection expose complex inter-service communication vulnerabilities (timeouts, retries, circuit breakers).
  • Carefully define blast radius, pre/post-conditions, and have robust observability and a rollback plan.

Code Example

bash
#!/bin/bash
NAMESPACE="default"
DEPLOYMENT_LABEL="app=my-service"

# Find a random running pod for the specified deployment
POD_NAME=$(kubectl get pods -n "${NAMESPACE}" -l "${DEPLOYMENT_LABEL}" \
           -o jsonpath='{.items[0].metadata.name}' \
           --field-selector=status.phase=Running 2>/dev/null)

if [ -z "$POD_NAME" ]; then
  echo "No running pods found for label '${DEPLOYMENT_LABEL}' in namespace '${NAMESPACE}'."
  exit 1
fi

echo "Initiating pod termination for: ${POD_NAME} in namespace ${NAMESPACE}"
kubectl delete pod "${POD_NAME}" -n "${NAMESPACE}" --wait=false

echo "Pod termination command sent. Monitor service health."

How this code works

This script performs a fundamental Chaos Engineering experiment: intentionally terminating a Kubernetes pod to observe how a system reacts to unexpected failures. It starts by defining NAMESPACE and DEPLOYMENT_LABEL to specify which group of pods to target. The kubectl get pods command then identifies one running pod matching these labels. It uses jsonpath to extract just the pod's name and --field-selector=status.phase=Running to ensure only active, healthy pods are considered for termination. If no suitable pod is found, the if [ -z "$POD_NAME" ] check gracefully exits, preventing errors.

Once a target POD_NAME is identified, the script proceeds with the "kill" action using kubectl delete pod. This command sends a signal to Kubernetes to remove the specified pod. A subtle but important detail for Chaos Engineering is the --wait=false option. This tells kubectl not to wait for the pod to fully terminate before returning control to the script. This ensures the script executes quickly, simulating a sudden, immediate failure and allowing the experiment to progress rapidly, rather than being delayed by the pod's graceful shutdown process. The goal is to induce chaos efficiently and observe the system's immediate recovery mechanisms.

Related Articles

Hand-picked reads to go deeper on chaos engineering.

Back to Site Reliability Engineer Roadmap