Application instrumentation is the act of embedding code within your application to expose internal metrics about its behavior and performance. For an SRE, this is the foundational layer of observability, providing the raw data Prometheus will scrape and Grafana will visualize. We primarily focus on three core metric types: Counters, Gauges, and Histograms. A Counter is a simple, monotonically increasing value, meaning it only ever goes up (or resets to zero on application restart). Use counters for tracking events like the total number of HTTP requests received, errors encountered, or bytes sent over a network. They are perfect for understanding cumulative activity, and when queried over time, can show rates (e.g., requests per second). In contrast, a Gauge represents a single numerical value that can go up or down at any point. Think of it as a thermometer or a speedometer, reflecting the current state. Examples include the current number of active users, available memory, CPU utilization, or the size of a processing queue.
While counters and gauges are excellent for simple counts and current states, they fall short when you need to understand the distribution of values, especially for critical performance metrics like request latency or response sizes. This is where Histograms shine. A histogram samples observations and groups them into configurable buckets (e.g., 0-10ms, 10-50ms, 50-100ms, etc.). Alongside these buckets, it also provides the total count of observations and their sum. The power of histograms lies in their ability to calculate percentiles (e.g., p99 latency) directly within Prometheus. This is crucial because averages can mask critical performance issues – a high average latency might hide that 1% of your users are experiencing very slow responses, something a histogram clearly reveals.
Effectively instrumenting your applications with these metric types is non-negotiable for building robust SRE practices. Prometheus client libraries (available for most languages) simplify exposing these metrics via a standard HTTP endpoint, which Prometheus then scrapes. In Grafana, you'll use Prometheus's powerful query language (PromQL) to turn these raw metrics into meaningful insights: calculating rates from counters, plotting current values from gauges, and deriving crucial latency percentiles from histograms. This direct feedback loop from your application's internals allows you to quickly identify performance bottlenecks, diagnose issues, and proactively improve system reliability, moving beyond simple "up/down" monitoring.
Key Takeaways
- Counters track monotonically increasing events (e.g., total requests, errors).
- Gauges track current state values that can go up or down (e.g., active users, CPU usage).
- Histograms track the distribution of observations (e.g., request latency) allowing for percentile calculations.
- Proper instrumentation is fundamental for deep observability into application behavior.
- Prometheus client libraries simplify exposing these metrics from your code.
Code Example
import (
"github.com/prometheus/client_golang/prometheus"
"time"
)
// Declare metrics using Prometheus client library
var (
requestsTotal = prometheus.NewCounterVec(
prometheus.CounterOpts{Name: "app_requests_total", Help: "Total requests."}, []string{"endpoint"})
activeConnections = prometheus.NewGauge(prometheus.GaugeOpts{
Name: "app_active_connections", Help: "Current active connections."})
requestLatency = prometheus.NewHistogramVec(
prometheus.HistogramOpts{Name: "app_request_latency_seconds", Help: "Request latency."},
[]string{"endpoint"})
)
func init() {
prometheus.MustRegister(requestsTotal, activeConnections, requestLatency)
}
// Example usage within your application logic
func handleAppEvent(endpoint string, duration time.Duration) {
requestsTotal.WithLabelValues(endpoint).Inc() // Increment counter
activeConnections.Inc() // Adjust gauge (could also Dec() or Set())
requestLatency.WithLabelValues(endpoint).Observe(duration.Seconds()) // Observe histogram
}
How this code works
This code instruments an application to emit crucial metrics for monitoring using the Prometheus client library. Its primary job is to declare and prepare three types of metrics: a counter for total requests, a gauge for current active connections, and a histogram for request latency. prometheus.NewCounterVec creates requestsTotal, designed to count events and categorize them by endpoint labels. prometheus.NewGauge defines activeConnections to track a value that fluctuates. prometheus.NewHistogramVec sets up requestLatency to measure the distribution of durations, also categorized by endpoint. The init() function automatically registers these metrics with the Prometheus system, making them discoverable and available for data collection.
The handleAppEvent function then demonstrates how an application's logic updates these metrics. requestsTotal.WithLabelValues(endpoint).Inc() increments the counter for a specific endpoint, tracking each request. activeConnections.Inc() adjusts the gauge, reflecting changes in active connections (it could also Dec() or Set() directly). For requestLatency.WithLabelValues(endpoint).Observe(), it's vital to note that histograms expect values in seconds, represented as a float64. Therefore, duration.Seconds() explicitly converts the time.Duration into the necessary format, preventing misinterpretation of latency values if other time units were implicitly used.