Phase 4: Incident Management

Rotation design: schedules, handoffs & coverage models

Intermediate ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine a really busy library that's open almost all the time. People always need help finding books, checking them out, or asking important questions. If only one librarian worked all the time, they'd get super tired and grumpy, and might miss important things or make mistakes. So, the library needs a smart way to make sure there's always a helpful librarian on duty, without anyone getting too exhausted. This is like setting up a special schedule for a team of librarians.

Think of it like different librarians taking turns to be the "on-duty" expert. For a whole week, maybe Ms. Daisy is the primary librarian, ready to answer any question or solve any problem. Then, for the next week, Mr. Tom takes over. This is called a "rotation." The library makes sure everyone gets fair turns, and enough time off to relax and read their own books. They also think about what time it is around the world; if a librarian is in a different country, their "shift" would happen when it's daytime for them, not the middle of the night! Special computer programs help plan these tricky schedules, making sure someone is always ready, but no one is overworked.

Now, imagine it's Friday afternoon, and Ms. Daisy's turn is almost over. Mr. Tom is starting his week on duty. This is a crucial moment called a "handoff." Ms. Daisy doesn't just run out the door! She tells Mr. Tom everything important: "Hey, that dragon book someone asked for is still missing," or "The new science books just arrived, you might want to put them on display," or "Remember that kid who needs help finding books about space? He usually comes in on Tuesdays." This way, when Mr. Tom starts his shift, he's immediately ready and knows exactly what's going on. He doesn't have to waste time figuring things out from scratch; he can keep helping people smoothly and keep the library running perfectly.

So, when you think about how important things like websites or games stay working all the time, even overnight, it's thanks to smart scheduling and careful handoffs just like in our library. This means you can build systems where a team of people can take turns making sure everything runs smoothly, sharing the responsibility and making sure no one person has to do everything all the time, keeping everyone fresh and ready to solve problems.

Designing effective on-call rotations is crucial for maintaining service reliability without burning out your team. It involves carefully structuring who is responsible for responding to incidents and when. A typical rotation assigns primary and sometimes secondary responders for defined periods, such as a week or a day. Key considerations include ensuring equitable distribution of shifts, taking into account time zones to align shifts with waking hours where possible, and building in sufficient off-call recovery time to prevent fatigue. Automated scheduling tools are indispensable here, allowing for complex rotations that balance team capacity with service demands.

Effective handoffs are the glue that holds a continuous on-call experience together. This is the process of transferring responsibility and critical context from the outgoing to the incoming on-call engineer. A structured handoff ensures the new engineer is immediately up to speed on any ongoing incidents, known issues, recent deployments, or pending follow-ups. This might involve a brief synchronous meeting, a dedicated chat channel for updates, or a shared document summarizing the state of affairs. The goal is to minimize friction and ensure a seamless transition of critical knowledge, so incidents don't lose momentum or context when a new person takes over.

Coverage models dictate how your organization ensures 24/7 availability. A common approach is a team-based rotation, where every member of a service's development or SRE team takes turns. For global organizations, a "follow-the-sun" model leverages teams in different time zones to hand off on-call duties, ensuring someone is always on-call during their local business hours, reducing night shifts. Another model involves tiered support, where a front-line operations team handles initial alerts (L1), escalating to a dedicated SRE or development team (L2) for deeper issues. The choice of model depends on team size, geographical distribution, service criticality, and the volume/complexity of incidents.

Key Takeaways

  • Design rotations to be fair and sustainable, preventing burnout with adequate off-call time and equitable shift distribution.
  • Implement clear, structured handoff processes to transfer incident context and ongoing issues seamlessly between shifts.
  • Utilize automated scheduling tools (e.g., PagerDuty, Opsgenie) to manage complex rotations and ensure proper coverage.
  • Choose a coverage model (e.g., team-based, follow-the-sun, tiered) that aligns with your team's size, location, and service criticality.
  • Prioritize continuous improvement of schedules and handoff procedures based on feedback and incident trends.

Code Example

yaml
# Simplified On-Call Schedule Configuration (e.g., PagerDuty)
# This defines a basic weekly rotation for a primary on-call team.

schedule:
  name: "Backend Service Primary On-Call"
  time_zone: "America/New_York"
  rotations:
    - name: "Primary Weekly Shift"
      type: "weekly" # Can also be 'daily', 'custom'
      start_time: "09:00:00" # HH:MM:SS
      users:
        - "sre.engineer.alpha"
        - "sre.engineer.beta"
        - "sre.engineer.gamma"
        - "sre.engineer.delta"
      # Layers or additional rotations can define secondary responders or specific time-of-day shifts.

How this code works

This code defines a simplified on-call schedule for a team, like what might be configured in an incident management system such as PagerDuty. Its main job is to establish who is responsible for primary on-call duties for a backend service and when their shifts occur. It ensures predictable coverage by clearly specifying the sequence and timing of on-call shifts, which is fundamental to a robust rotation design, enabling smooth handoffs and continuous service availability.

The top-level schedule block sets up the overall framework with a descriptive name and specifies the time_zone for accurate shift timings, crucial for geographically distributed teams. Within this, the rotations section describes the actual on-call shifts. Here, a type of weekly is chosen, meaning each user listed under users will take a full week-long shift before the next person in the list takes over. A subtle but important detail for beginners is that the system will iterate through the users list sequentially. If sre.engineer.alpha is on call this week, sre.engineer.beta will be on call the next week by default, ensuring a consistent rotation. The start_time indicates when that specific weekly shift officially begins.

Related Articles

Hand-picked reads to go deeper on on-call practices.

Back to Site Reliability Engineer Roadmap