Phase 4: Incident Management

On-call tooling: pagers, dashboards & quick-access runbooks

Intermediate ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you're the super important head chef at a busy pizza restaurant, but you can't be in the kitchen watching everything all the time. Sometimes you need to step out to get more ingredients, or even go home for the night! But what if the big pizza oven suddenly gets too hot and starts to burn the pizzas? Or what if the special dough mixer breaks down right in the middle of a rush? This is where your super-smart "on-call tooling" comes in.

First, you have a special "chef's alert system" – kind of like a special, loud phone or pager that only rings for real emergencies, not just if someone needs more cheese. This system is always watching all the kitchen's important parts, like the oven's temperature or the mixer's speed. If something goes wrong, like the oven getting dangerously hot, this system knows who the head chef on duty is (maybe it's you this week!) and calls or sends a message immediately. It makes sure you hear about the problem super fast, even if you're asleep, so you can run back to the kitchen and fix it before too many pizzas are ruined.

Once you get that urgent alert and rush into the kitchen, you don't want to just stare at the smoking oven and guess what happened. That's where your "kitchen control center" comes in – it's a giant screen covered in helpful charts and pictures. This screen shows you everything that's happening in real-time: the oven's temperature history, how much power the mixer is using, how many pizzas are being made, if the fridges are cold enough, and even how many customers are waiting. It helps you quickly see if it's just the oven, or if the fridge is also broken, or if maybe someone accidentally turned up the heat too high. It's like the kitchen's brain scan, showing you all its vital signs at a glance so you can figure out the problem quickly.

And finally, if you see the oven is too hot, you don't have to remember every single step to fix it while everyone is yelling for their pizza. You have a special, easy-to-find "Emergency Fix-It Book." This book has super clear, step-by-step instructions for common problems, like "If the oven is too hot, check this dial first, then open that vent, and if it's still hot, call the oven repair company." These tools work together to help you be the best head chef ever, ready to keep the kitchen running smoothly no matter what. This means you can build amazing things, knowing you have a superhero backup plan to fix problems before they become huge disasters.

Being on-call means being ready to respond to system incidents, and effective tooling is the backbone of a successful response. Pagers are your first line of defense, integrating with monitoring systems (like Prometheus or Datadog) to alert the correct on-call engineer based on predefined schedules and escalation policies. When a critical metric breaches a threshold or an error rate spikes, the pager system ensures you get notified immediately via phone call, SMS, or mobile app push notifications. These systems are crucial for minimizing Mean Time To Acknowledge (MTTA) by cutting through noise and ensuring urgent issues don't go unnoticed, even in the middle of the night.

Once an alert has been acknowledged, dashboards become indispensable for understanding the incident's scope and potential root cause. These visual interfaces aggregate real-time metrics, logs, and traces from across your infrastructure and applications, presenting them in an easily digestible format. A well-designed dashboard allows an on-call engineer to quickly identify trends, pinpoint affected services, view error rates, and track resource utilization. Tools like Grafana, Kibana, or Datadog provide the ability to create bespoke dashboards that offer a comprehensive 'single pane of glass' view, helping you move from 'what' is happening to 'where' it's happening much faster.

Finally, quick-access runbooks provide the 'how-to' for diagnosing and resolving common incidents. These are documented, step-by-step procedures that guide engineers through common operational tasks or incident response workflows. Often linked directly from an alert or dashboard, runbooks reduce cognitive load during stressful situations, standardize responses, and ensure consistency across the on-call team. They capture valuable institutional knowledge, detailing commands to run, logs to check, common failure patterns, and escalation paths, ultimately helping to reduce Mean Time To Resolution (MTTR) by providing immediate, actionable guidance rather than relying on guesswork or tribal knowledge.

Key Takeaways

  • Pagers are critical for timely alert escalation and ensuring on-call engineers are notified effectively.
  • Dashboards provide immediate visual context and diagnostic information, aiding in quick problem identification.
  • Quick-access runbooks standardize incident response, reduce stress, and improve resolution times.
  • These tools work synergistically: pagers alert, dashboards diagnose, and runbooks guide resolution.
  • Regularly review and update runbooks to keep them accurate and useful.

Code Example

bash
# A simple runbook snippet to check service status and recent logs
#!/bin/bash

SERVICE_NAME="api-gateway"
LOG_PATH="/var/log/application"

echo "--- Checking systemctl status for ${SERVICE_NAME} ---"
systemctl status ${SERVICE_NAME} | head -n 7

echo "\n--- Displaying last 20 ERROR logs for ${SERVICE_NAME} ---"
grep -i "ERROR" ${LOG_PATH}/${SERVICE_NAME}.log | tail -n 20

echo "\n--- Attempting a basic health check via HTTP ---"
curl -s -o /dev/null -w "HTTP Status: %{http_code}" http://localhost:8080/health

How this code works

This runbook snippet provides a quick way to check the health and recent activity of a critical service, like an api-gateway, during an on-call incident. It begins by setting up SERVICE_NAME and LOG_PATH for easy customization. First, the script uses systemctl status to get a summary of the service's current operational state, piping its output through head -n 7 to show only the most relevant first few lines. Next, grep -i "ERROR" searches the application's log file for any recent "ERROR" messages, displaying the last twenty occurrences using tail -n 20. The -i flag ensures it finds errors regardless of their capitalization.

Finally, the script performs a basic health check using curl against a local health endpoint. The curl command is specifically configured with -s to run silently, -o /dev/null to discard the actual page content (since only the status matters), and -w "HTTP Status: %{http_code}" to print only the HTTP status code. This focused output is a subtle but important choice, preventing a flood of irrelevant data and making it clear at a glance whether the service is responding with a healthy status, which might otherwise be buried in verbose curl output if these options weren't used.

Related Articles

Hand-picked reads to go deeper on on-call practices.

Back to Site Reliability Engineer Roadmap