Phase 4: Incident Management

Writing effective postmortems: timeline, impact & root cause

Intermediate ~4 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Sometimes, even the coolest things we build, like an amazing LEGO castle or a super-fast race car made of blocks, accidentally fall apart or don't work quite right. When that happens, it’s not about getting upset or figuring out who to blame. Instead, it's about being like a detective! We want to figure out exactly what went wrong so we can learn from it and build something even stronger and more stable next time. This is what grown-up engineers call a "postmortem" – a fancy word for looking back after something has happened to understand it better.

Imagine your incredible LEGO tower unexpectedly toppled over. The first thing you'd want to do is remember exactly what happened, step by step, like replaying a video in your mind. This is called creating a timeline. You'd think: "First, I carefully placed the big green baseplate. Then, I stacked the tall red bricks really high. Oh, wait, I remember my little sister bumped the table just as I was trying to balance that wobbly yellow flag on top! And then it crashed down!" You'd write down the order of everything, maybe even when things happened (like "right after dinner" or "while watching that cartoon"). You wouldn't blame your sister; you'd just note the events in a clear, factual way so everyone understands the story.

After that, you'd think about the impact. How big was the mess? Did all your carefully built sections scatter everywhere? Did it take a long time to clean up and sort the bricks? Did it ruin the surprise you were planning for your parents? Understanding the impact helps you see how important it is to fix the underlying problem. Finally, you'd try to figure out the root cause. Why did it really fall? Was the base not wide enough for such a tall tower? Was the flag piece too heavy for the thin walls supporting it? Was the table itself a bit wobbly to begin with? It wasn't just your sister's bump; it was probably a combination of things that made the bump have such a big effect. The "root cause" is like finding the real, underlying reason, not just the last thing that happened.

By writing down this clear story – the timeline, the impact, and the root cause – you can show your friends or even your future self: "Look, this is why my tower fell last time. So this time, I'm going to build a wider, sturdier base, use lighter pieces for the top, and maybe even build on a really solid table!" This means when you grow up and start building complicated systems and software for computers, you'll know how to learn from mistakes, make things much stronger, and prevent similar problems from happening again, just like a super smart LEGO master builder.

Writing an effective postmortem is a critical skill for Site Reliability Engineers, transforming incidents from mere disruptions into valuable learning opportunities. The postmortem isn't about assigning blame but rather about understanding what happened, quantifying its impact, and identifying systemic issues to prevent recurrence. To achieve this, a clear, factual timeline is paramount. This chronological account should detail every significant event: when the incident was detected, the initial symptoms, the actions taken by responders (including specific commands or configurations changed), escalations, and finally, the time of resolution. Precision in timestamps and a neutral, objective description of events are crucial, often reconstructed from logs, monitoring alerts, and communication channels like chat transcripts or incident tickets.

Following the timeline, you must clearly articulate the impact of the incident. This involves quantifying how users, systems, or business operations were affected. For user-facing services, this might include metrics like increased error rates (e.g., 5xx errors), elevated latency, or complete downtime, along with the number of affected users or transactions. For internal systems, the impact could be on developer productivity, degradation of dependent services, or missed internal SLAs. Wherever possible, use hard data and metrics – duration of impact, specific error counts, or even estimated financial loss – to provide a clear picture of the incident's severity and scope. This objective measurement helps prioritize future preventative work.

Finally, and perhaps most critically, an effective postmortem delves into the root cause. This goes beyond merely identifying the proximate cause (what immediately triggered the incident) to uncover the underlying systemic vulnerabilities. Instead of just saying "a server rebooted," you'd ask why it rebooted unexpectedly, why the failover didn't work, and why monitoring didn't alert correctly. Root causes often point to deficiencies in monitoring, process gaps, architectural design flaws, missing runbooks, or inadequate testing. The goal is to identify actionable items that address these fundamental weaknesses, ensuring that the same type of incident is far less likely to happen again. Remember to focus on systems and processes, not individuals.

Key Takeaways

  • Timeline: Reconstruct a precise, chronological sequence of events using objective data.
  • Impact: Quantify the incident's effect on users and systems with specific metrics.
  • Root Cause: Identify systemic vulnerabilities and process gaps, not just immediate triggers or individual actions.
  • Focus on learning and actionable prevention, avoiding blame.
  • Be objective and data-driven throughout the postmortem.

Code Example

bash
# Example: Basic log search to help reconstruct an incident timeline
# In a real SRE context, you'd typically use dedicated logging platforms (ELK, Splunk, Grafana Loki, etc.)

START_TIMESTAMP="2023-10-26 10:00:00" # Example incident start time
END_TIMESTAMP="2023-10-26 10:30:00"   # Example incident end time
SERVICE_LOG_FILE="/var/log/my_app/service.log"
SEARCH_TERMS="ERROR|latency|timeout|deployment|rollback|restart"

# Filter log entries within the incident window containing relevant terms
# NOTE: This assumes log timestamps are sortable strings at the beginning of each line.
grep -E "$SEARCH_TERMS" "$SERVICE_LOG_FILE" \
  | awk -v ts_start="$START_TIMESTAMP" -v ts_end="$END_TIMESTAMP" '{
      # Extract the timestamp part assuming format like "YYYY-MM-DD HH:MM:SS"
      log_timestamp = substr($0, 1, length(ts_start));
      if (log_timestamp >= ts_start && log_timestamp <= ts_end) {
        print $0
      }
    }' \
  | sort -k1,2 # Sort by date and time to ensure chronological order

How this code works

This code snippet's primary job is to filter a large SERVICE_LOG_FILE to pinpoint specific events that occurred during an incident, defined by START_TIMESTAMP and END_TIMESTAMP. This helps reconstruct a precise timeline of what happened, an essential step in postmortem analysis. The script first sets up crucial parameters: START_TIMESTAMP and END_TIMESTAMP define the time window of interest, SERVICE_LOG_FILE specifies which log to examine, and SEARCH_TERMS lists keywords like ERROR or timeout to look for. These variables ensure the subsequent commands focus only on relevant data during the critical period.

The filtering process begins with grep -E. This command acts as an initial sieve, quickly pulling out any log lines that contain any of the specified SEARCH_TERMS. The output of grep is then piped to awk. The awk command performs a more refined time-based filter, checking if the log entry's timestamp (extracted using substr($0, 1, length(ts_start))) falls within the ts_start and ts_end window. A subtle but important detail is awk's reliance on length(ts_start). This assumes log line timestamps exactly match the format and length of the defined START_TIMESTAMP string at the very beginning of each line, which can trip up beginners if log formats differ. Finally, sort -k1,2 arranges the remaining filtered entries chronologically by date and time, presenting a clear, ordered incident timeline.

Related Articles

Hand-picked reads to go deeper on postmortems & learning.

Back to Site Reliability Engineer Roadmap