Phase 5: Platform Engineering

Cluster operations: upgrades, node management & etcd backups

Advanced ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you're building an amazing LEGO city. Not just one small house, but a whole big city with lots of different buildings: a fire station, a hospital, a zoo, and even a spaceport! This entire city, with all its connected parts working together, is like a "cluster" in the world of computers. Just like you want your LEGO city to always be fun and working perfectly for your mini-figures, grown-ups want their computer systems to always run smoothly for everyone using them.

Sometimes, you get new, super cool LEGO sets or instructions that make your city even better or stronger. Maybe the fire station gets a new helicopter pad, or the hospital gets an emergency room expansion! This is like "upgrading" your city. You wouldn't just smash the old fire station to build the new one, right? That would be chaos! Instead, you carefully move the tiny firefighters to a temporary station, build the new one, and then move them back. This careful process is how we handle "upgrades" in a computer system – we update parts one by one without shutting down the whole thing. And what if one of your buildings needs fixing, or you want to add a brand new one, or even remove an old one? This is like "managing your nodes" (your buildings). You wouldn't just pull a building out of the middle of your city if it had people inside! You'd gently move all the mini-figures out first, make sure they have somewhere else to go, and then you can safely work on or move that building.

Now, imagine you have a super important instruction manual for your entire LEGO city. It tells you exactly where every single brick goes for every building, how the roads connect, and where all your mini-figures live. If you lost that manual, and someone accidentally knocked over half your city, you'd have no idea how to rebuild it perfectly! This super important manual is like the "etcd" (pronounced et-see-dee) in a computer cluster. It’s the master blueprint for everything. If we lose the etcd, we lose the whole city – poof! That's why grown-ups always make copies of this super important manual. They might take photos of every page, or scan it into a computer file, and keep these copies in a super safe place. This is called "backing up" the etcd. By regularly making these copies, even if the main manual gets lost or damaged, we can always use a copy to rebuild our amazing LEGO city exactly how it was, making sure everything works perfectly again.

So, when you hear about grown-ups doing "cluster operations," they're basically being the super careful city planners for huge computer systems. They make sure everything runs smoothly, gets updated safely, and that there's always a backup plan for the most important instructions. This means that all the apps and websites you use, like your favorite online games or streaming videos, can stay online and work perfectly almost all the time, even when the grown-ups are making big changes behind the scenes!

Kubernetes cluster operations demand a robust strategy for maintaining high availability. Upgrades, a frequent necessity, require a meticulous rolling update approach for both the control plane and worker nodes to minimize service disruption. SREs must understand the upgrade path, potential API deprecations, and utilize tools like kubeadm upgrade or managed service equivalents. Before touching worker nodes, applications should be gracefully migrated. Node management involves safely adding, removing, or performing maintenance. Critical tools here are kubectl cordon to prevent new pods from scheduling on a node, and kubectl drain to evict existing pods gracefully, ensuring workloads are rescheduled onto healthy nodes. Automating node lifecycle events, especially with cloud provider auto-scaling groups, is crucial for elastic and resilient infrastructure.

The etcd key-value store is the single source of truth for your Kubernetes cluster state; losing it means losing your cluster. Therefore, robust and regularly tested etcd backup and restore procedures are paramount for disaster recovery. For self-managed clusters, this typically involves taking snapshots of the etcd data directory or using etcdctl snapshot save from within an etcd container or host. Cloud-managed Kubernetes services often handle etcd backups internally, but SREs must still understand and verify their recovery point objectives (RPOs) and recovery time objectives (RTOs). Crucially, backups must be stored securely off-cluster and the restoration process thoroughly documented and regularly practiced in a non-production environment to ensure successful recovery during an actual incident.

From an SRE perspective, these operations are not merely tasks but critical components of a reliable system. Every upgrade or node operation should be treated as a potential incident, necessitating comprehensive monitoring, alerting, and well-defined rollback plans. Automation is key to reducing human error and ensuring consistency across environments. Leveraging infrastructure as code (IaC) for node configuration and deployment, combined with continuous integration/continuous deployment (CI/CD) pipelines for rolling out changes, significantly enhances operational safety. Ultimately, the goal is predictable, automated, and observable cluster health, allowing SREs to proactively manage the platform rather than reactively troubleshoot failures.

Key Takeaways

  • Adopt rolling strategies for upgrades, prioritizing control plane stability and minimal downtime.
  • Master kubectl drain and kubectl cordon for safe, graceful node maintenance and removal.
  • Implement and regularly test automated etcd backup and restoration procedures; it's your cluster's lifeline.
  • Automate node lifecycle management using tools like autoscalers for elasticity and resilience.
  • Embrace observability, automation (IaC, CI/CD), and runbooks for all cluster operations.

Code Example

bash
# 1. Safely cordon and drain a worker node for maintenance
kubeclt cordon my-worker-node-01
kubeclt drain my-worker-node-01 --ignore-daemonsets --delete-emptydir-data --force --grace-period=120

# 2. Perform a robust etcd backup (from an etcd container in kube-system)
# First, identify an etcd pod
ETCD_POD=$(kubectl get pods -n kube-system -l component=etcd,k8s-app=etcd -o jsonpath='{.items[0].metadata.name}')
# Execute snapshot command within the pod
kubeclt exec -it -n kube-system "$ETCD_POD" -- etcdctl snapshot save --endpoints=127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key /var/lib/etcd/snapshot.db
# Important: Copy the snapshot.db out of the pod to persistent, off-cluster storage
# Example: kubeclt cp -n kube-system "$ETCD_POD":/var/lib/etcd/snapshot.db ./etcd-backup-$(date +%F).db

How this code works

This code demonstrates essential SRE operations for managing a Kubernetes cluster: safely preparing a worker node for maintenance and performing a robust etcd backup. First, it prepares a worker node named my-worker-node-01. The kubeclt cordon command prevents new pods from being scheduled onto this node. Subsequently, kubeclt drain gracefully evicts all existing pods from it. Key flags like --ignore-daemonsets ensure essential infrastructure pods are not moved, while --delete-emptydir-data cleans up temporary local data, and --grace-period=120 provides applications ample time to shut down, making the node ready for safe maintenance.

The second part focuses on backing up etcd, Kubernetes's central data store. ETCD_POD=$(kubectl get pods ...) dynamically identifies an etcd pod by its labels in the kube-system namespace, making the command resilient to pod name changes. Then, kubeclt exec ... etcdctl snapshot save runs the etcd backup command directly inside that pod, securely creating a snapshot.db file. A subtle but crucial point for beginners is that this snapshot is created inside the etcd pod's temporary filesystem. The final commented line, kubeclt cp ..., highlights the critical need to copy this snapshot.db file out of the pod to persistent, off-cluster storage, otherwise, the backup will be lost when the pod is eventually terminated.

Resources

Related Articles

Hand-picked reads to go deeper on kubernetes for sre.

Back to Site Reliability Engineer Roadmap