Phase 4: Incident Management

Incident roles: commander, communicator & subject-matter experts

Intermediate ~2 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you're playing a really important soccer game, and suddenly things start going wrong. Maybe your team can't score, or the other team is getting too many chances. If everyone on your team just runs around, trying to do everything at once – trying to defend, attack, and maybe even tell the referee what's happening – it would be super chaotic, right? Nobody would know what to focus on, and the problems would just get worse. This is exactly why, even when computers have big problems (we call these "incidents"), we need clear jobs for everyone involved.

On a computer team that fixes problems, one super important job is like the Team Captain or the Coach on the sidelines during a crucial moment. This person is called the Incident Commander. Their main job isn't to "kick the ball" themselves (do the hands-on fixing). Instead, they look at the whole "game," figure out the biggest problem, and decide the best strategy. They tell everyone what to focus on and make sure the right players are doing their specific jobs. The players who are actually kicking the ball, making amazing passes, or blocking shots are like the Subject-Matter Experts. These are the people who are really, really good at one specific part of the computer system, and they do the actual work to get it back to normal, following the Commander's plan.

But wait, there's another super important job! What about the fans in the stands, or the referee, or even the coach for the next game? They need to know what's happening! That's where the Incident Communicator comes in. This person is like the team's spokesperson. They make sure everyone outside the immediate huddle – like the fans, or other teams, or even other parts of your own company – knows what's going on and what's being done to fix the problem. They make sure the message is clear and honest, so nobody gets confused or worried.

So, when you're older and building your own cool games or websites, and something goes wrong, you'll know that having clear roles – a leader to strategize, experts to fix things, and a communicator to keep everyone informed – is the best way to solve problems quickly and smoothly. Just like a well-organized soccer team wins matches, a well-organized incident response team gets computer systems working perfectly again!

When a critical incident strikes, chaos can quickly ensue without clear leadership and division of labor. This is where defined incident roles come into play, providing structure and ensuring that crucial tasks are not overlooked or duplicated. For an SRE, understanding these roles isn't just theoretical; it's about efficiently restoring service, minimizing impact, and learning from the event. These roles aren't necessarily fixed titles but functions that need to be filled for effective incident management, ensuring that technical resolution, stakeholder communication, and overall incident strategy are all actively managed.

The Incident Commander (IC) is the overall owner and orchestrator of the incident. Their primary responsibility is strategic: assessing the situation, setting priorities, defining the next steps, and delegating tasks to technical experts. The IC doesn't necessarily perform the hands-on fixes but ensures that the right people are doing the right things, manages the incident timeline, and makes executive decisions. Alongside the IC, the Incident Communicator (ICom) acts as the single source of truth for all external (and often internal) stakeholders. The ICom's job is to shield the technical team from constant inquiries, craft clear and timely updates for customers, executives, and other affected teams, and ensure consistent messaging throughout the incident lifecycle.

Finally, Subject-Matter Experts (SMEs) are the technical specialists brought in to diagnose and resolve the specific problem. These individuals could be database administrators, network engineers, application developers, or any other expert whose knowledge is critical to understanding and fixing the outage. SMEs focus entirely on the technical problem-solving, reporting their findings and progress to the Incident Commander. While these are distinct roles, in smaller incidents, a single SRE might temporarily wear multiple hats (e.g., act as IC and an SME), but the critical functions of command, communication, and technical resolution must always be covered.

Key Takeaways

  • Incident roles provide structure and prevent chaos during critical outages.
  • The Incident Commander (IC) leads the incident, focusing on strategy, delegation, and overall resolution.
  • The Incident Communicator (ICom) manages all external and internal stakeholder communication, shielding the technical team.
  • Subject-Matter Experts (SMEs) provide critical technical skills for diagnosis and hands-on resolution.
  • Roles are functional and dynamic; one person might fulfill multiple roles in smaller incidents, but the functions must always be covered.

Code Example

yaml
incident:
  id: "P0-20231027-001"
  type: "Database Unavailability"
  severity: "P0"
  status: "active_investigation"
  active_roles:
    commander: "@sre-alice"
    communicator: "@comms-bob"
    smes:
      - "@db-charlie" # Database SME
      - "@infra-diana" # Infrastructure SME
  timeline:
    - timestamp: "2023-10-27T10:00:00Z"
      event: "Incident declared by @sre-frank"
    - timestamp: "2023-10-27T10:05:00Z"
      event: "Roles assigned and bridge opened"

How this code works

This YAML structure serves as a clear, human-readable record of an ongoing incident, specifically outlining its key attributes and, most importantly, the individuals currently assigned to critical response roles. The top-level incident key contains details like the id, type (Database Unavailability), and severity (P0), giving an immediate overview of the situation. Its status is active_investigation, indicating the problem is currently being addressed.

The active_roles section is where the core lesson is illustrated, defining who is doing what. The commander is @sre-alice, overseeing the entire response, while @comms-bob is the communicator, handling updates to stakeholders. A subtle but important detail is the smes key, which is a list allowing for multiple Subject Matter Experts. Unlike the single commander or communicator, an incident can often require diverse expertise, as shown with @db-charlie (Database SME) and @infra-diana (Infrastructure SME) contributing concurrently. The timeline then chronicles significant events with their precise timestamps, building a chronological log of incident actions.

Related Articles

Hand-picked reads to go deeper on incident response.

Back to Site Reliability Engineer Roadmap