Phase 5: Platform Engineering

Deployment safety: canary releases, blue-green & progressive rollouts

Intermediate ~4 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you have a beautiful garden with all your favorite flowers and vegetables growing perfectly. Everyone loves visiting! You have an idea for a new kind of super-cool flower, or a better way to water everything. The big worry is: what if your new idea accidentally makes all your old, favorite plants wilt or messes up your whole garden for everyone who visits?

To avoid that, gardeners have clever ways to try out new things safely. One way is like having two identical gardens right next to each other. Your first, perfect garden is like the "Blue" garden – it's the one everyone sees now. Then, you create a completely new, separate "Green" garden. In this "Green" garden, you plant all your new flowers or try out your new watering system. You make sure everything works perfectly there, hidden from view. Once you're absolutely sure your "Green" garden is amazing and ready, you just open up the gate to the "Green" garden and everyone starts looking at it instead of the old "Blue" one. If, by some small chance, something goes wrong in your "Green" garden, you can instantly close its gate and open the "Blue" garden's gate again. Crisis averted! Everyone just goes back to enjoying the original, safe garden.

Another way is even more careful, like trying a new fertilizer. Instead of putting it on all your plants in your new "Green" garden, you pick just a couple of special plants, maybe some very strong, healthy ones, to be your "canaries." You put the new fertilizer on only these few canary plants. You watch them closely. If they thrive, you know the fertilizer is good! You then slowly, carefully, put it on a few more plants, until eventually, all your plants have the new fertilizer. This "progressive rollout" means you're watching every step of the way. If even one canary plant starts to look sad, you stop immediately, and only a tiny part of your garden was affected. This way, you can slowly introduce new ideas without risking your whole beautiful garden.

So, when people build and update websites or apps, they use these strategies to bring you cool new features and fix bugs without anything breaking. This means you can always expect your favorite apps to work, knowing that the people who build them are using smart, safe ways to make them better, one careful step at a time!

Deployment safety is paramount for Site Reliability Engineers, ensuring new code reaches production without causing outages or negative user impact. High-stakes environments demand strategies that minimize risk and facilitate rapid recovery. Blue-green deployment is one such technique, involving two identical, independent production environments: "Blue" (the current stable version) and "Green" (the new version). Traffic is routed entirely to Blue initially. Once the new Green environment is fully deployed and tested internally, the load balancer or DNS is atomically switched to direct all incoming traffic to Green. If any critical issues arise post-switch, a quick rollback simply means switching traffic back to the stable Blue environment. This method offers fast rollbacks but can be resource-intensive, requiring double the infrastructure for a brief period.

Canary releases offer a more granular approach to risk mitigation. Instead of an all-or-nothing switch, a small subset of user traffic (the "canary") is directed to the new version ("Green") while the majority still uses the stable "Blue" version. This allows real-world performance and error rates of the new version to be monitored with minimal exposure. If the canary performs well against defined SLOs and health metrics, traffic is progressively increased to the new version, often in stages (e.g., 5%, 20%, 50%, 100%). Should issues surface, the canary traffic can be immediately redirected back to the stable version, isolating the problem to a small user group. This method significantly reduces blast radius but requires robust monitoring and can slow down the full deployment process.

"Progressive rollouts" serve as an umbrella term for any deployment strategy that introduces new code gradually to production. Both blue-green (if you consider the switch a very fast 0-100% progression) and especially canary releases fall under this category. Beyond simple traffic weighting, progressive rollouts can involve deploying to specific user segments, geographic regions, or internal teams first. The core principle is to manage risk by incrementally exposing changes and continuously validating their impact before wider adoption. The choice between these methods depends on factors like application architecture, acceptable downtime, rollback speed requirements, and infrastructure costs. Implementing these requires sophisticated CI/CD pipelines, robust monitoring, and automated health checks.

Key Takeaways

  • Blue-Green deployments offer instant rollback by switching between two full, identical environments but require duplicate infrastructure.
  • Canary releases minimize risk by gradually exposing a new version to a small user subset, demanding strong real-time monitoring.
  • Progressive Rollouts is a broader strategy for phased deployment, encompassing methods like canary, to reduce the impact of potential issues.
  • All these strategies aim to reduce the blast radius of faulty deployments and increase confidence in production releases.

Code Example

bash
# Assume 'my-app-v1' is running with 10 replicas and 'my-app-v2' is a new deployment (initially 0 replicas).
# A service would typically point to both via shared labels or an ingress controller manages traffic weighting.

# Phase 1: Canary - Scale v2 to 1 replica (10% traffic), v1 to 9 replicas
kubectl scale deployment/my-app-v2 --replicas=1
kubectl scale deployment/my-app-v1 --replicas=9

# (Monitor v2 health and performance extensively)

# Phase 2: Progressive rollout - Scale v2 to 5 replicas (50% traffic), v1 to 5 replicas
kubectl scale deployment/my-app-v2 --replicas=5
kubectl scale deployment/my-app-v1 --replicas=5

# (Monitor again, if stable, proceed)

# Phase 3: Full rollout - Scale v2 to 10 replicas (100% traffic), v1 to 0 replicas
kubectl scale deployment/my-app-v2 --replicas=10
kubectl scale deployment/my-app-v1 --replicas=0

# Clean up old version after full confidence (optional)
kubectl delete deployment my-app-v1

How this code works

This code demonstrates a "canary release" strategy, a safe way to deploy a new version of an application (my-app-v2) without immediately affecting all users. The goal is to gradually introduce my-app-v2 to a small percentage of traffic, monitor its performance, and then progressively roll it out to everyone if stable. It starts with my-app-v1 handling all traffic and my-app-v2 initially having no instances running.

The process unfolds in phases using kubectl scale. First, "Phase 1: Canary" uses kubectl scale deployment/my-app-v2 --replicas=1 to bring up a single instance of the new version. Simultaneously, kubectl scale deployment/my-app-v1 --replicas=9 reduces the old version's instances, directing a small portion (10%) of traffic to my-app-v2. A crucial, subtle point here is that kubectl scale only adjusts instance counts; the actual traffic weighting (like 10% vs 90%) is handled by an external component like a Kubernetes Service or Ingress Controller, which automatically distributes requests across available replicas. If the canary performs well, "Phase 2: Progressive rollout" scales both my-app-v2 and my-app-v1 to 5 replicas each, balancing traffic 50/50. Finally, "Phase 3: Full rollout" scales my-app-v2 to its full 10 replicas and my-app-v1 to 0, ensuring all users are on the new version before my-app-v1 is eventually removed with kubectl delete deployment.

Resources

Related Articles

Hand-picked reads to go deeper on ci/cd reliability.

Back to Site Reliability Engineer Roadmap