Phase 2: Observability

Prometheus architecture, scraping & service discovery

Intermediate ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you have a huge garden with many different kinds of plants: tall sunflowers, tiny strawberries, busy rose bushes, and fast-growing herbs. To make sure everything grows well, you need to know how each plant is doing – are the strawberries getting enough sun? Is the rose bush looking healthy? This is like needing to know what's happening inside the many computer programs that make up a website or an app. It's too much for one person to remember or look at all the time!

That's where Prometheus (let's call it "Farmer Promi") comes in. Instead of each plant shouting, "I need water! I need sun!", Farmer Promi walks through the garden. Every so often, Farmer Promi stops at each plant, looks at it, and writes down important details like "Strawberry bush has 5 berries" or "Sunflower is 7 feet tall." These details are called "metrics." Each plant has a special little tag on it, like a tiny sign, that says, "Hey, Farmer Promi, here's where you can find my health update!" This act of Farmer Promi going to each plant and collecting its information is called "scraping." So, Farmer Promi pulls the information from the plants, rather than the plants pushing it.

Sometimes, a plant might be super tiny and only pop up for a very short time, like a mushroom after rain. It might disappear before Farmer Promi gets a chance to visit. For these fast-growing "jobs," there's a special little mailbox in the garden. Before the mushroom disappears, it quickly drops its information into this mailbox. Then, when Farmer Promi does their rounds, they check the mailbox too, and collect all those quick updates. These "plant tags" and "mailboxes" are like special helpers called "exporters," or sometimes they're even part of the plant itself, making sure the info is always ready for Farmer Promi.

So, Farmer Promi collects all these measurements and puts them in a special notebook, noting the time of each observation. Later, you can look at this notebook and see how the strawberries grew over a week, or if a certain rose bush always looks droopy in the afternoon. This means you can spot problems early, understand what makes your garden (or your computer programs!) healthy, and help everything run smoothly so your apps and websites work perfectly for everyone.

Prometheus functions on a pull model, where the Prometheus server actively scrapes metrics endpoints from configured targets at regular intervals. Its core architecture involves a time-series database for storing scraped data, an HTTP server for exposing its own metrics and API, a scraper component, and a rule processing engine for alerts and recording rules. Applications and infrastructure components don't push metrics to Prometheus; instead, they expose an HTTP endpoint (typically /metrics) where Prometheus can fetch them. This is often achieved using 'exporters' (e.g., Node Exporter for host metrics, cAdvisor for container metrics) or by instrumenting applications directly with client libraries. For short-lived jobs, a Pushgateway can act as an intermediary, allowing transient services to push metrics to it, which Prometheus then scrapes.

The process of scraping is fundamental: Prometheus reads its scrape_configs from prometheus.yml, which define jobs, intervals, and target endpoints. For each target, it makes an HTTP request to its /metrics path, retrieves the data in Prometheus text format (or OpenMetrics), and stores it as time-series data, adding relevant labels. Each scraped metric includes a timestamp, a value, and a set of key-value pairs called labels that uniquely identify the characteristics of that metric (e.g., instance="webserver-01", job="api-service"). This label-based data model is incredibly powerful for querying and aggregating metrics.

In dynamic environments like Kubernetes clusters or cloud deployments, manually configuring every scrape target in prometheus.yml is impractical and error-prone. This is where service discovery comes in. Prometheus integrates with various service discovery mechanisms (e.g., Kubernetes API, AWS EC2, Consul, DNS) to automatically discover and track scrape targets. Instead of static IP addresses, you configure Prometheus to query a service discovery backend. This backend then provides a list of potential targets, which are then processed by Prometheus's relabeling rules. Relabeling allows you to dynamically modify target labels, filter targets, or even alter the scrape endpoint path based on metadata provided by the service discovery system, ensuring Prometheus always knows what to scrape, even as your infrastructure scales and changes.

Key Takeaways

  • Prometheus uses a pull model, actively scraping metrics from targets.
  • Targets expose metrics via HTTP endpoints (e.g., /metrics), often facilitated by exporters.
  • Service discovery automates finding and tracking targets in dynamic environments.
  • Relabeling rules are crucial for transforming discovered targets and their metadata into valid scrape configurations.

Code Example

yaml
scrape_configs:
  - job_name: 'node_exporter'
    # Example of static targets, useful for fixed infrastructure.
    # In a dynamic environment, service discovery would generate these.
    static_configs:
      - targets: ['localhost:9100', 'server-02:9100']
        labels:
          group: 'test'
    # Defines how often Prometheus should scrape this job.
    scrape_interval: 15s
    scrape_timeout: 10s
  
  # Example of a Kubernetes service discovery config (simplified).
  # Prometheus automatically discovers pods/services based on roles.
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    # Relabeling rules would typically follow here to filter, rewrite labels,
    # and set the correct __address__ for scraping based on pod metadata.

How this code works

This configuration defines how Prometheus discovers and collects metrics from various services. The first section sets up a scrape_configs entry with job_name: 'node_exporter'. It uses static_configs to explicitly list targets like localhost:9100 and server-02:9100, which is ideal for fixed infrastructure. These targets also get a label group: 'test' for easy organization. scrape_interval: 15s tells Prometheus to try fetching data every 15 seconds, and scrape_timeout: 10s ensures it stops waiting for a response after 10 seconds, preventing hung scrapes.

The second scrape_configs entry, job_name: 'kubernetes-pods', demonstrates dynamic service discovery. It uses kubernetes_sd_configs with role: pod, instructing Prometheus to automatically discover all running pods in a Kubernetes cluster by watching its API. A subtle but crucial point for beginners is that while Prometheus discovers these pods, the example mentions that relabeling rules would typically follow. These rules are essential for filtering specific pods, rewriting labels, and especially for setting the correct __address__ for Prometheus to actually scrape the pod's metrics endpoint, as the raw discovery data isn't always directly scrapable.

Related Articles

Hand-picked reads to go deeper on monitoring with prometheus & grafana.

Back to Site Reliability Engineer Roadmap