Phase 5: Platform Engineering

Kubernetes observability: metrics-server, kube-state-metrics & events

Advanced ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you're the super important manager of a really busy restaurant. You have lots of cooks (which are like little computer programs called "pods") and big ovens, fridges, and work tables (which are like bigger computers called "nodes"). To make sure customers get their food on time and everything runs smoothly, you need to know exactly what's happening. This is called "observability"—being able to see and understand everything that goes on.

One of your main jobs is to make sure no cook gets too overwhelmed. You have a special little assistant, let's call them the "Cook Monitor." (This is like metrics-server.) The Cook Monitor quickly goes around to each cook, peeking over their shoulder to see how much chopping and stirring they're doing (how much "CPU" or brainpower they're using) and how much counter space they've filled up with ingredients (how much "memory" or temporary storage they need). This assistant gives you a quick, real-time update. So, if you ask, "Who's the busiest cook right now?" (like using a special command called kubectl top pod), the Cook Monitor tells you instantly. This helps you decide if you need to automatically call in an extra cook when things get really crazy busy.

But knowing how busy cooks are isn't enough. You also have another assistant, let's call them the "Kitchen Status Reporter." (This is kube-state-metrics.) This reporter doesn't care about how fast a cook is chopping. Instead, they keep track of the status of everything important in the kitchen. Is oven number 3 turned on? Is the fridge door closed? Has table 5 been cleaned and is it ready for new customers? Is that special pizza dough currently being made, or has it finished rising? The Kitchen Status Reporter listens to all the goings-on and tells you if things are "ready," "waiting," "off," or "broken." This helps you understand the condition of your kitchen's tools and services, not just how hard people are working.

And finally, sometimes things happen suddenly! A cook might accidentally spill a tray of soup, or the big ingredient delivery truck might pull up, or a customer at table 2 just finished their dessert. These are like little shouts or quick notes called "events." They tell you that something just happened. You get a quick alert: "Soup spill in aisle 3!" or "New lettuce just arrived!" These messages are important because they let you react right away to unexpected situations or important updates, even if they're not a big "problem" or a constant "status" report. So, by having your Cook Monitor, Kitchen Status Reporter, and Event Shouts all working together, you, the manager, can keep your restaurant running perfectly. This means you can build really smart systems that keep your computer programs happy and your users delighted, without you having to constantly watch every single thing yourself!

Kubernetes observability for SREs hinges on understanding the cluster's health, performance, and operational state. Three foundational components provide crucial insights: metrics-server, kube-state-metrics, and events. metrics-server is a cluster-wide aggregator of resource usage data. It collects CPU and memory metrics from Kubelets on each node, making this information available via the Kubernetes API. Its primary role isn't for long-term storage but to power built-in features like kubectl top node and kubectl top pod, and, critically, to enable Horizontal Pod Autoscalers (HPA) and Vertical Pod Autoscalers (VPA) to make scaling decisions based on actual resource consumption. Without metrics-server, your autoscaling capabilities based on resource usage are severely limited, and real-time performance diagnostics become significantly harder.

While metrics-server tells you how much resource a pod or node is consuming, kube-state-metrics tells you what state Kubernetes objects are in. It's a service that listens to the Kubernetes API server and generates metrics about the state of various objects like Deployments, Pods, Nodes, PersistentVolumes, and more. Instead of resource usage, it exposes metrics such as kube_deployment_status_replicas_available, kube_pod_status_phase (e.g., Running, Pending, Failed), or kube_persistentvolumeclaim_status_phase. These metrics are exposed in Prometheus format, making them invaluable for understanding the overall health, capacity, and operational status of your cluster. SREs leverage kube-state-metrics for building dashboards that monitor deployment rollout status, identify stuck pods or volumes, track node readiness, and generally understand the "health" of the control plane and workloads from a logical, rather than resource-usage, perspective.

Finally, Kubernetes events provide a chronological log of changes and occurrences within the cluster. Whenever a pod is scheduled, a container starts or fails, a volume is attached, or a node goes offline, an event is generated. These are short-lived records, typically retained for only an hour or so by default, making them crucial for real-time debugging of transient issues. An SRE investigating why a pod isn't starting would immediately check events associated with that pod or its owning deployment to see scheduling failures, image pull errors, or volume attachment problems. While ephemeral, events are an indispensable first-line diagnostic tool, providing context for why something is in a particular state. Together, these three pillars form a comprehensive picture: metrics-server for resource consumption, kube-state-metrics for object state, and events for understanding the dynamics and lifecycle changes that lead to those states.

Key Takeaways

  • metrics-server provides real-time CPU/memory usage metrics for pods and nodes, essential for kubectl top and HPA/VPA.
  • kube-state-metrics exposes Kubernetes object state (e.g., pod phases, deployment replicas) as Prometheus metrics, crucial for logical cluster health and capacity planning.
  • Events offer a short-lived, chronological log of cluster activities and state changes, vital for immediate troubleshooting and understanding lifecycle issues.
  • These components form a complementary observability foundation, covering resource usage, object state, and the dynamics that drive those states.

Code Example

bash
# Check for recent cluster-wide warning events
kubectl get events --all-namespaces \
  --field-selector type=Warning \
  --sort-by='.lastTimestamp' \
  -o custom-columns="LAST SEEN:.lastTimestamp,TYPE:.type,REASON:.reason,OBJECT:.involvedObject.kind/.involvedObject.name,MESSAGE:.message" \
  --limit 5

How this code works

This command gathers a filtered list of warning events from across a Kubernetes cluster, presenting them in a structured, readable format. Its purpose is to help SREs quickly review potential issues by showing significant events that might indicate problems across different components of the system.

The command begins by fetching all Kubernetes events using kubectl get, expanding its scope to all-namespaces for a cluster-wide view. It then applies a crucial filter with field-selector type=Warning, narrowing down the results to focus solely on events categorized as warnings. These often signal potential underlying issues. The results are then ordered using sort-by='.lastTimestamp', which, by default, sorts in ascending chronological order. This is a subtle point that might trip up a beginner: despite aiming to find "recent" warnings, limit 5 will, in this setup, display the five earliest observed warning events rather than the most current ones. Finally, the -o custom-columns option formats the output, defining custom column headers like LAST SEEN and REASON and extracting specific data points such as involvedObject.kind and message to make the critical event details easy to review.

Resources

Related Articles

Hand-picked reads to go deeper on kubernetes for sre.

Back to Site Reliability Engineer Roadmap