Phase 4: Incident Management

Writing clear, step-by-step runbooks for common scenarios

Beginner ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you’re in the kitchen, and you want to bake your favorite chocolate chip cookies. But oh no! You’ve never baked them before, or maybe you're super sleepy. Without a recipe, you might grab the wrong ingredients, guess at the oven temperature, and end up with something... well, not quite cookies! That’s kind of what it’s like for the grown-ups who build and look after all the websites and apps you use. Sometimes, things go a bit wrong – a game might slow down, or a video might not load. When that happens, it’s like a kitchen disaster! But instead of a burned cake, it’s a computer problem. To fix it quickly and without panicking, they use something called a "runbook."

A runbook is basically a super clear, step-by-step instruction manual, just like your favorite cooking recipe. If you have a recipe for chocolate chip cookies, it tells you exactly what to do: "First, get 2 cups of flour. Second, add 1 cup of sugar. Third, mix them together in a big bowl." It even tells you how long to bake them and at what temperature. You don't have to guess, and anyone can follow it to make delicious cookies, even if they're new to baking!

In the world of computers, if a website suddenly stops working, the engineers don't just stare at it and wonder. They look for the "website not working" runbook. This runbook is like a recipe for fixing that specific problem. It might say: "Step 1: Check if the main computer is on. Step 2: Look at the error messages in this special log file. Step 3: Try restarting the website's software with this command." Just like a recipe makes sure your cookies turn out great every time, a runbook helps make sure computer problems get fixed quickly and correctly, no matter who is doing the fixing or how sleepy they are.

So, writing clear, step-by-step runbooks means breaking down a big, tricky computer problem into tiny, easy-to-follow steps. It’s like breaking down a complicated cookie recipe into simple actions. This skill means that even when you start building your own amazing websites or games in the future, you’ll know how important it is to write down clear instructions. This way, if something unexpected happens, you and anyone helping you will know exactly what to do to get things running smoothly again, just like baking perfect cookies every time.

As an SRE, incidents are inevitable. Runbooks are your secret weapon for dealing with them calmly and effectively. Think of a runbook as a detailed, step-by-step recipe for resolving specific common issues you might encounter in your infrastructure. When an alert fires at 3 AM, or a user reports a specific error, a clear runbook prevents panic and guesswork. It ensures everyone on the team can follow a consistent process, leading to faster diagnosis and resolution, which significantly reduces your Mean Time To Restore (MTTR) critical services. A well-written runbook translates complex tribal knowledge into actionable, reproducible instructions for anyone on call.

Writing clear, step-by-step runbooks means breaking down a problem into its smallest, most manageable actions. Start by identifying common scenarios – for example, a "disk full" error, a "database connection refused," or a "service not starting." For each scenario, outline the process chronologically: begin with the initial symptoms (what the alert or user reports), then guide the user through diagnostic steps (where to check logs, metrics, or service status), and finally, provide precise resolution steps (commands to run, configurations to modify, or services to restart). Use numbered lists and bullet points for readability. Crucially, each step should be unambiguous, explaining not just what to do, but also what to expect as an outcome or what indicators to look for to confirm success or failure.

Beyond just listing steps, a good runbook includes essential context like prerequisites (e.g., "SSH access to server X," "Kubernetes CLI installed"), links to relevant dashboards or monitoring tools, and clear escalation paths if the runbook doesn't resolve the issue. Keep the language simple and direct, avoiding jargon where possible, as runbooks are often used under pressure by various team members, including junior engineers. Always test your runbooks by having someone unfamiliar with the issue follow them. Finally, treat runbooks as living documents: store them in an easily accessible, version-controlled system (like a wiki or Git repository) and commit to regular review and updates. As your systems evolve, so too must your runbooks to remain effective.

Key Takeaways

  • Runbooks are step-by-step guides for common incidents, helping teams respond quickly and consistently.
  • Structure runbooks chronologically: symptoms -> diagnosis -> resolution, with clear, unambiguous steps.
  • Each step should describe what to do, how to do it, and what to expect (e.g., command output).
  • Include prerequisites, links to tools, and clear escalation paths for unresolved issues.
  • Test runbooks with others, store them accessibly (version-controlled), and update them regularly.

Code Example

bash
# Example Runbook Step: Check Web Server Status and Recent Logs

echo "--- Step 1: Check the status of the 'nginx' service ---"
systemctl status nginx

echo ""
echo "--- Step 2: If nginx is not 'active (running)', attempt to restart it ---"
read -p "Do you want to restart nginx? (y/N): " restart_choice
if [[ "$restart_choice" =~ ^[yY]$ ]]; then
    sudo systemctl restart nginx
    echo "nginx restart initiated. Checking status again..."
    sleep 5
    systemctl status nginx
else
    echo "nginx restart skipped."
fi

echo ""
echo "--- Step 3: Review the last 20 lines of the nginx error log ---"
tail -n 20 /var/log/nginx/error.log

How this code works

This script provides a structured runbook step for diagnosing a potential issue with an Nginx web server. Its primary job is to first assess the server's operational status, then offer a guided option to attempt a fix if necessary, and finally present relevant log data for further investigation. This sequence helps SREs efficiently follow a process to check, troubleshoot, and gather information about common web server scenarios.

The code initiates by executing systemctl status nginx to display the current state of the Nginx service. Following this, it employs read -p to present an interactive prompt, asking if a restart is desired. A subtle yet important detail is the (y/N) in the prompt: if a user simply presses Enter without typing anything, the script's default behavior is to not restart, indicated by the capital 'N'. If 'y' or 'Y' is provided, sudo systemctl restart nginx is run, and a sleep 5 pause ensures the service has a moment to start before systemctl status nginx checks its state again. The final step uses tail -n 20 /var/log/nginx/error.log to output the last 20 lines of the Nginx error log, offering immediate diagnostic insight.

Related Articles

Hand-picked reads to go deeper on runbooks & documentation.

Back to Site Reliability Engineer Roadmap