Phase 5: Monitoring, Observability & Reliability

Custom Application Metrics

Intermediate ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you're helping run a very busy, popular bakery. You want to make sure everything is going well. One way you check is by looking at things like if the lights are on, if the ovens have power, or if the refrigerator is cold enough. These are super important because if the power goes out, you can’t bake anything! In computer terms, this is like checking if your computer server (the big machine running your bakery app) has enough memory or if its main "brain" (CPU) isn't overworked. It tells you if the building itself is okay.

But just knowing the building is okay doesn't tell you if your bakery is actually successful or if customers are happy. What if the ovens are on, but all the cakes are burnt? Or if you have plenty of flour, but you're only selling one cookie a day? This is where "custom application metrics" come in. They’re like special little notes you write down inside your baking process. For example, you might want to track: How many chocolate chip cookies did we bake today? How long did it take, on average, for a customer to get their fresh bread after ordering? How many cakes are currently waiting to be decorated? Or how many times did someone accidentally put salt instead of sugar in the cookies?

These specific, detailed notes are your custom metrics. They focus on the important parts of what your bakery actually does – baking and selling delicious treats – not just whether the lights are on. To do this with a computer program, a programmer would add tiny instructions directly into the program’s recipe (its code). So, when your app successfully signs up a new user, it makes a note, "New User Count: +1!" Or, when a customer buys something, it starts a timer and records how long the payment process took. These instructions use special tools, kind of like having a special clipboard and pen just for these important notes.

By looking at these custom notes later, you can really understand how your bakery (or computer program) is doing. If you see that it's taking too long for customers to get their bread, you can investigate – maybe the bread slicer is broken, or a new baker needs more practice. If the "salt-instead-of-sugar" count suddenly goes up, you know you need to check that recipe! So, when you build your own amazing apps, these custom metrics mean you can make sure your app is not just running, but actually working well and making users happy by keeping track of the little things that matter most in its daily operations.

When monitoring applications, you often track infrastructure-level metrics like CPU usage, memory consumption, or disk I/O. While essential, these don't tell you what's happening inside your application's business logic. Custom application metrics fill this gap. They are specific, domain-aware data points that you instrument directly within your application code to expose its internal state and behavior. Think about metrics like the number of successful user registrations, the average latency of your checkout API, the queue depth of pending messages, or the count of failed login attempts. These are critical for understanding the actual health, performance, and operational efficiency of your application from a user or business perspective, going beyond just knowing if the server is alive.

Implementing custom metrics, especially with Prometheus, involves instrumenting your application's source code using a Prometheus client library (available for most major programming languages like Python, Go, Java, Node.js, etc.). These libraries provide simple interfaces to define and update various metric types – such as counters (for cumulative events), gauges (for current values), and histograms/summaries (for observing distributions of values like request durations). Your application then exposes these metrics via a standard HTTP endpoint, typically /metrics, which Prometheus can scrape at regular intervals. This means developers integrate the monitoring directly into the features they build, ensuring that key operational data is always available.

The value of custom application metrics is immense for a DevOps Engineer. They transform raw data into actionable insights, enabling you to detect subtle issues that infrastructure metrics would miss. For instance, a server might have low CPU, but custom metrics could reveal a spike in failed payment transactions or a slowdown in a critical background job. With these metrics, you can create highly specific Grafana dashboards to visualize business-critical flows, set intelligent alerts based on application-specific thresholds (e.g., "if checkout success rate drops below 95%"), and perform detailed root cause analysis. They bridge the gap between application development and operations, offering a holistic view of your system's performance and reliability.

Key Takeaways

  • Expose internal application state and business logic.
  • Instrument application code directly using Prometheus client libraries.
  • Provide domain-specific insights beyond generic infrastructure metrics.
  • Crucial for advanced observability, debugging, and targeted alerting.

Code Example

python
from prometheus_client import Counter, Gauge, start_http_server
import time, random

# Define custom metrics
REQUEST_COUNTER = Counter('my_app_requests_total', 'Total requests to the application.')
ACTIVE_CONNECTIONS = Gauge('my_app_active_connections', 'Number of active client connections.')
PROCESS_TIME = Gauge('my_app_process_time_seconds', 'Time taken to process last request.')

def simulate_work():
    REQUEST_COUNTER.inc()
    ACTIVE_CONNECTIONS.inc()
    start_time = time.time()
    time.sleep(random.uniform(0.05, 0.2)) # Simulate work
    PROCESS_TIME.set(time.time() - start_time)
    ACTIVE_CONNECTIONS.dec()

if __name__ == '__main__':
    # Start server to expose metrics on http://localhost:8000/metrics
    start_http_server(8000)
    print("Prometheus metrics server started on port 8000")

    # Update metrics in a loop
    while True:
        simulate_work()
        time.sleep(1)

How this code works

This Python code creates a simple "application" that generates custom metrics, making it observable by monitoring systems like Prometheus. It demonstrates how to instrument an application to expose internal state, allowing insights into its behavior. The code begins by importing necessary tools from prometheus_client and time. It then defines three specific metrics: REQUEST_COUNTER as a Counter to track the total number of incoming requests, ACTIVE_CONNECTIONS as a Gauge to show the current number of active processes, and PROCESS_TIME also as a Gauge to record the duration of the last completed task. Finally, start_http_server(8000) starts a lightweight web server, which will make these defined metrics available for Prometheus to collect at http://localhost:8000/metrics.

The core logic resides in the simulate_work function, which mimics processing a request. Inside this function, REQUEST_COUNTER.inc() increments the total request count, and ACTIVE_CONNECTIONS.inc() increases the active connection count. A brief pause using time.sleep simulates actual work. After the simulated work, PROCESS_TIME.set() records the time taken for just this specific request. A subtle point here is that PROCESS_TIME always reflects the duration of the most recently completed operation, not an average or cumulative sum. Finally, ACTIVE_CONNECTIONS.dec() decrements the active connection count, indicating the work is complete and the connection is no longer active. The program runs this simulation repeatedly within a while True loop, continuously updating the exposed metrics.

Related Articles

Hand-picked reads to go deeper on monitoring & metrics (prometheus / grafana).

Back to DevOps Engineer Roadmap