Phase 3: Reliability Engineering

Game days: planned failure exercises for team readiness

Advanced ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

You know how sometimes at school, a fire alarm will ring, but it's not a real fire? It’s just a fire drill! Everyone knows it's a drill because your teachers told you it would happen, or you heard about it beforehand. The whole point of a fire drill isn't because the school actually wants there to be a fire. Instead, it’s a practice session, a way for everyone to learn exactly what to do, where to go, and how to stay safe if a real fire ever happened. It helps everyone act calmly and quickly.

Well, imagine that important computer systems, like the ones that run your favorite online games or streaming videos, are a lot like a busy school. They need to be super reliable and always working. So, the grown-ups who build and look after these systems have something similar to a fire drill, which they call a "Game Day." On a Game Day, they plan to make a small part of their computer system act like it's broken, just for a little while. They're not trying to actually break anything important forever, but rather to see how everyone involved — the people who write the code, the people who watch for problems, and the people who fix things — reacts to a pretend emergency.

During this planned computer "failure," the team watches carefully. Do their alarms go off when something breaks? Do they notice the problem quickly? Do they know exactly what steps to take to fix it? Can they talk to each other efficiently, like teachers coordinating during a drill? For example, they might pretend that the computer's "address book" for finding specific users suddenly stops working. Then, they watch how their team works together to find the issue, fix it, and make sure everything is running smoothly again. Just like you might discover during a fire drill that a hallway is blocked, they might find a communication tool isn't working or that a step in their "fix-it" plan is missing.

So, when you're older and start building your own games or websites, you'll know that practicing for when things go wrong is a super powerful idea. It means you can make your creations much more robust and reliable, ensuring that when a real problem pops up, you and your team will be ready to tackle it like pros!

In Site Reliability Engineering, a 'Game Day' is a structured, planned exercise designed to simulate real-world infrastructure failures or adverse conditions. Unlike ad-hoc chaos experiments that might target specific technical vulnerabilities, game days primarily focus on evaluating and improving the human and process elements of incident response. The goal isn't just to break things, but to observe how your team responds under pressure, how effective your monitoring, alerting, runbooks, and communication channels are, and ultimately, to build muscle memory and confidence for actual outages. They are collaborative events involving SREs, development teams, and sometimes even product owners, all participating in a controlled environment.

The practical execution of a game day involves meticulous planning. This includes defining clear objectives, selecting a specific failure scenario (e.g., a database replica going down, an API dependency becoming unavailable, a specific AWS region failing), and establishing a precise scope and blast radius. Safety mechanisms like a 'kill switch' to immediately halt the experiment are paramount. During the exercise, teams actively engage in incident detection, diagnosis, mitigation, and resolution, following their established on-call procedures. Post-exercise, a critical debrief session identifies shortcomings across tooling, documentation, team coordination, and knowledge gaps. This feedback loop is essential for driving actionable improvements.

Game days move beyond merely identifying technical defects; they expose systemic weaknesses in your operational readiness. They help validate hypotheses about system resilience and, more importantly, about team performance during stress. By routinely practicing these planned failures, organizations can refine their incident response playbooks, enhance observability, improve cross-team communication, and ensure that when a real incident strikes, the team is well-rehearsed, confident, and capable of restoring service efficiently. Ultimately, game days are an investment in your team's ability to maintain high reliability.

Key Takeaways

  • Game days are planned, team-wide exercises simulating failures to test human and process readiness.
  • They focus on incident response, communication, runbooks, and team coordination, not just technical bugs.
  • Meticulous planning, clear objectives, safety mechanisms (kill switch), and a post-exercise debrief are crucial.
  • Identify and address gaps in tooling, documentation, and team processes for continuous improvement.
  • Builds team confidence and muscle memory, significantly improving real-world incident handling.

Code Example

yaml
# A conceptual YAML definition for a Game Day scenario using a generic chaos platform
---
apiVersion: chaos.example.com/v1alpha1
kind: ChaosExperiment
metadata:
  name: db-replica-failure-gameday
  labels:
    gameday: 'true'
spec:
  scenarioName: "Database Replica Failure Simulation"
  description: "Simulate the graceful termination of a single DB replica to test failover and incident response."
  target:
    service: "payment-service-database"
    instanceSelector: "role=replica,az=us-east-1a"
    count: 1 # Target a single replica
  action: "terminate_process"
  duration: "30m" # 30 minutes of simulated impact
  hypothesis:
    - "Automatic DB failover will complete within 5 minutes."
    - "Payment service p99 latency will not exceed 200ms during failover."
  observability:
    metricsToCheck:
      - "db_replica_count_gauge"
      - "payment_service_latency_ms"
  safety:
    killSwitchURL: "https://chaos-dashboard.example.com/kill/db-failover"
    rollbackProcedure: "docs/db-failover-rollback-procedure.md"
  participants:
    - "sre-team-alpha"
    - "dev-team-payments"

How this code works

This YAML code defines a "Game Day" exercise, specifically a ChaosExperiment named db-replica-failure-gameday. Its job is to simulate a controlled failure in a production-like environment to test how a team and system react to unexpected issues. The scenarioName and description provide human-readable context, outlining a plan to intentionally cause a problem and observe the results, ensuring the system can handle events like a database replica failing gracefully.

Within the spec, the experiment details are laid out. The target specifies the payment-service-database, focusing on a single replica in a particular availability zone. The action is terminate_process, simulating a sudden crash, lasting for a duration of 30m. Crucially, the hypothesis section details expected outcomes, such as automatic failover completing quickly and payment latency remaining stable – these aren't executable code but define the success criteria for the Game Day. For safety, the killSwitchURL and rollbackProcedure ensure the experiment can be stopped or reverted if things go wrong. A subtle but vital detail is count: 1 within target, which explicitly limits the impact to just one replica, preventing an accidental full outage.

Related Articles

Hand-picked reads to go deeper on chaos engineering.

Back to Site Reliability Engineer Roadmap