Phase 2: Observability

Grafana dashboards, variables & annotations

Intermediate ~2 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you have a super special recipe book that shows you exactly how well everything in your kitchen is doing! Each recipe page is like a "dashboard" – it gives you a quick, easy-to-understand picture of something important. One page might show your oven's temperature, another tracks cookies baked today, or how much milk is left. It’s your control center, helping you keep an eye on all things delicious at a glance.

Here's the clever part! Let's say you have a fantastic cupcake recipe. You might want chocolate sometimes, vanilla other times, or to add sprinkles. Instead of needing a new recipe for each, your special book has a trick. Right at the top of the cupcake page, there are "chooser" buttons. Pick "chocolate" or "vanilla." The moment you click, the recipe instantly updates to show exactly how to make those specific cupcakes! These choosers are like "variables." They let you use one main recipe page to see many versions of your kitchen's performance without creating endless pages. This keeps your book tidy and helps you quickly compare, for example, "chocolate" cupcake sales versus "vanilla" ones.

But what if something unexpected happens? Maybe your famous cookie recipe suddenly starts turning out too crumbly. Your recipe book has another secret helper: "annotations." These are like little sticky notes you can place right onto a recipe page, marking a specific day and time. For example, you might write, "May 5th: Started using a new type of flour!" Then, if your cookies become crumbly later, you can look at the page, see your note, and realize, "Aha! It might be because of that new flour!" These notes give you important clues about why things change, helping you quickly figure out what happened.

So, with these smart tools, you can keep your kitchen super organized and quickly understand what's happening. You can easily switch between viewing your "dinner menu" performance versus "dessert menu" using those choosers. And if you see something surprising, like an ice cream machine slowing down, you can quickly spot sticky notes to see if you made any changes. This means you can quickly figure out problems and keep everything running smoothly!

Grafana dashboards are your operational single pane of glass, visualizing metrics from Prometheus. To make these dashboards truly powerful and flexible for an SRE, Grafana's variables are indispensable. Variables allow you to create dynamic dashboards where users can select parameters like environment, service, host, or namespace from dropdowns. Instead of building separate dashboards for development, staging, and production, or for each microservice, you can use a single templatized dashboard. This drastically reduces dashboard sprawl, simplifies maintenance, and accelerates incident investigations by letting you quickly pivot views without changing the dashboard itself. Common variable types include query (fetching values from your data source, e.g., Prometheus label_values), custom (hardcoded options), and textbox (free-form input).

Beyond dynamic views, Grafana annotations provide crucial context directly on your graphs. Annotations are markers, typically vertical lines with descriptive text, that pinpoint significant events in time. For an SRE, this means you can visually correlate metrics spikes or dips with events like application deployments, configuration changes, or major incidents. Imagine seeing a latency increase and, overlaid on the same graph, an annotation marking a recent code deployment – this immediately helps narrow down the cause. Annotations can be added manually, fetched from a data source via a Prometheus query (e.g., up{job="my-service"} offset 5m), or integrated from external systems like CI/CD pipelines, incident management tools, or change logs, providing an invaluable timeline for root cause analysis.

By mastering Grafana dashboards, variables, and annotations, you transform static monitoring panels into an interactive, diagnostic powerhouse. Variables empower you to build reusable dashboards that adapt to any service or environment, cutting down on repetitive work and ensuring consistency. Annotations provide the critical "why" behind the "what" you see in your metrics, making it easier to connect observed behavior with operational events. Together, they are fundamental tools for proactive monitoring, efficient incident response, and thorough post-mortem analysis, enabling SREs to maintain reliable systems with greater precision and speed.

Key Takeaways

  • Variables enable dynamic, reusable dashboards, significantly reducing dashboard sprawl.
  • Annotations provide critical context by marking significant events directly on time-series graphs.
  • Together, variables and annotations enhance the ability to correlate metrics with operational events.
  • They streamline incident investigations by allowing quick filtering and visual event identification.
  • Transforms static dashboards into powerful, interactive tools for proactive monitoring and post-mortem analysis.

Code Example

json
{
  "templating": {
    "list": [
      {
        "name": "job",
        "type": "query",
        "dataSource": "Prometheus",
        "query": "label_values(up{job!=\"\"}, job)",
        "multi": true,
        "includeAll": true,
        "label": "Prometheus Job"
      }
    ]
  },
  "annotations": {
    "list": [
      {
        "name": "Deployments",
        "datasource": "Prometheus",
        "enable": true,
        "query": "deployment_events{status=\"success\"}",
        "step": "1h"
      }
    ]
  }
}

How this code works

This code defines elements for a Grafana dashboard, specifically setting up a dynamic filtering dropdown and adding event markers to the timeline. The templating section creates a variable named job which powers a dropdown menu labeled "Prometheus Job." Its query fetches all unique job labels from Prometheus, but importantly, job!="" ensures that only non-empty, meaningful job names are presented in the filter, preventing irrelevant blank options. The multi: true setting allows users to select multiple jobs from the dropdown, while includeAll: true adds a convenient "All" option to view data across every job.

The annotations section overlays event markers on the dashboard's graphs, visualizing significant occurrences like successful deployments. It uses the Prometheus datasource and a query: "deployment_events{status=\"success\"}" to retrieve metrics that indicate successful deployments. The step: "1h" configuration is key here; it tells Grafana to group any events occurring within the same hour into a single annotation marker. This prevents the timeline from becoming overwhelmed with numerous individual markers if many deployments happen close together, providing a cleaner, summarized view of activity.

Related Articles

Hand-picked reads to go deeper on monitoring with prometheus & grafana.

Back to Site Reliability Engineer Roadmap