Phase 2: Observability

PromQL queries: rates, aggregations & recording rules

Intermediate ~2 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you’re playing a super cool board game with lots of turns and ways to earn points. You could just count your total points at the end, but that doesn't really tell you how you’re doing right now. Are you scoring points quickly? Are you stuck? Instead, we need a special way to see how fast things are changing. Think of a clever referee who watches the game. This referee can look at how many points you’ve scored over the last five turns, and then tell you your average points per turn. If you’re getting about 10 points every five turns, that's a smooth, steady number that helps you see if you're generally doing well or badly over time. This is perfect for understanding long-term trends, like your overall progress in the game.

But what if something exciting just happened? Maybe you just played an amazing card that earned you a ton of points in the very last second! The referee has another way to watch for sudden changes. They can look only at the points you scored between the very last two score updates. If there’s a huge jump right then, they’d immediately shout, "Whoa, big change now!" This second way is super quick at noticing big, sudden spikes or drops, like a rapid burst of points or if you suddenly stopped earning any. It’s like the referee blowing the whistle instantly for a foul or a fantastic play.

Now, picture your game has not just one player, but many players, maybe even different teams playing on different parts of the board! Each player has their own score, their own "points per turn," and their own sudden jumps. That's a lot of numbers to keep track of, and it can get confusing quickly. Sometimes, you don't need to know every single player's detailed score. You might just want to know how many points the whole Red Team is scoring per turn, or the average score of all players in the "Forest Zone" of the board.

This is where we "aggregate" or group the scores. The referee can add up all the Red Team's points per turn to give you one total number for the Red Team. Or they could find the average points per turn for everyone playing in the "Forest Zone." By doing this, you can easily understand the bigger picture, like which team is winning overall, or which part of the game board is getting the most action, without getting lost in all the tiny details of every player. This means you can make smart decisions, like sending more players to a struggling part of the board, or knowing exactly when to celebrate a sudden winning streak!

When analyzing time-series data in Prometheus, raw counter values are often less useful than their rate of change. This is where rate() and irate() come in. rate(my_counter[5m]) calculates the average per-second increase of my_counter over the last 5 minutes, ideal for stable, long-term trends like requests per second (RPS) or bandwidth usage. irate(my_counter[5m]), on the other hand, specifically looks at the instantaneous rate over the last two data points within that 5-minute window. This makes irate() more sensitive to sudden spikes or drops, making it particularly useful for rapid-fire alerts where quick detection of change is critical. Understanding which one to use depends on whether you need a smoothed average or immediate responsiveness to recent changes.

Prometheus metrics often have many labels (e.g., instance, job, status_code, method), making raw query results very granular. Aggregation operators like sum(), avg(), max(), min(), and count() allow you to summarize this data across different dimensions. For instance, sum(rate(http_requests_total[5m])) by (job) will give you the total RPS for each job, effectively rolling up data from multiple instances within that job. Conversely, sum(rate(http_requests_total[5m])) without (instance) would sum RPS for each job and status_code, dropping the instance label. These aggregations are fundamental for creating meaningful dashboards and extracting higher-level insights, transforming raw data into actionable information by reducing noise and focusing on key dimensions.

As your monitoring setup grows, complex PromQL queries can become slow and resource-intensive, especially when used repeatedly across multiple dashboards or alerts. Recording rules are Prometheus's solution for this. They allow you to define a PromQL expression and have Prometheus pre-compute its result at regular intervals, saving it as a new time series. For example, you can define a rule to calculate sum(rate(http_requests_total[5m])) by (job) every minute. This pre-computed metric (e.g., job:http_requests_total:rate5m) can then be queried much faster by Grafana dashboards or alerting rules, significantly reducing the query load on your Prometheus server and improving the responsiveness of your observability stack. They are crucial for building high-performance, resilient monitoring systems.

Key Takeaways

  • rate() and irate() transform raw counters into meaningful rates of change, with rate() for averages and irate() for responsiveness.
  • Aggregation operators (sum, avg, max, etc.) with by() or without() reduce data granularity for actionable insights.
  • Recording rules pre-calculate expensive or frequently used PromQL expressions, creating new, faster-to-query metrics.
  • Combine these techniques to build efficient, performant dashboards and alerts.
  • Leverage recording rules to reduce Prometheus query load and improve UI responsiveness.

Code Example

yaml
groups:
  - name: application-overview
    rules:
      - record: job:http_requests_total:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))
      - record: instance:node_cpu_usage:avg1m
        expr: 100 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[1m])) * 100

How this code works

This code defines Prometheus "recording rules," which serve to pre-calculate the results of frequently used or complex PromQL queries and store them as new, simpler metrics. This approach significantly speeds up dashboard loading times and simplifies alert rule definitions, as dashboards and alerts can query these pre-computed metrics directly instead of re-evaluating complex expressions repeatedly. Specifically, these rules create a new metric named job:http_requests_total:rate5m to track the 5-minute rate of HTTP requests, aggregated by job, and another named instance:node_cpu_usage:avg1m for the 1-minute average CPU usage percentage, broken down by instance.

The groups block organizes these rules. For the job:http_requests_total:rate5m metric, its expr uses rate(http_requests_total[5m]) to calculate the per-second rate of increase of the http_requests_total counter over a five-minute window, then sum by (job) aggregates these rates across all instances belonging to the same job. For instance:node_cpu_usage:avg1m, the expr calculates the rate of node_cpu_seconds_total specifically in "idle" mode over one minute, averages it by instance, and then subtracts this idle percentage from 100 to yield the actual CPU usage percentage. A subtle point for beginners is that rate() always returns a per-second rate, regardless of the [duration] (e.g., [5m]) specified; the duration merely defines the time window over which to observe the change.

Related Articles

Hand-picked reads to go deeper on monitoring with prometheus & grafana.

Back to Site Reliability Engineer Roadmap