Phase 4: Architecture & Scaling

Alerting rules & anomaly detection

Intermediate ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you have a super important garden, like a really special one filled with magical plants that help you power your amazing projects. You want to make sure your plants are always happy and healthy!

One way to keep them healthy is to set up "alerting rules." These are like simple instructions for yourself. For example, you might have a rule that says: "If the soil for my Sparklebloom plant feels dry for more than two days, I need to water it!" Or "If I see more than three weeds growing near the Rainbow-root, I need to pull them out!" These rules are great because they tell you exactly what to look for and what to do when something obvious goes wrong. You decide ahead of time what counts as a problem (like "too dry" or "too many weeds"), and as soon as that specific thing happens, you get a little reminder, like a note taped to your watering can. This helps you fix common problems quickly, before your Sparklebloom gets too thirsty.

But sometimes, problems aren't so obvious. That's where "anomaly detection" comes in. Think of it like having a super-smart plant-observing robot friend for your garden. This robot doesn't just follow simple rules. Instead, it spends lots of time learning what each plant usually looks like when it's perfectly happy. It remembers how quickly the Rainbow-root usually grows, how green its leaves normally are, or what time of day the Sparklebloom usually opens its flowers. After a while, it knows what "normal" looks like for every single plant.

Then, if something unusual happens – something that doesn't fit the normal pattern, even if it's not something you made a specific rule for – your robot friend notices! Maybe the Sparklebloom's leaves are a tiny bit less shiny than usual, or the Rainbow-root is growing just a little slower this week, even though the soil isn't dry and there aren't any weeds. It's not a disaster yet, but it's different. Your robot friend would then give you a gentle nudge, "Hey, the Sparklebloom is behaving a little differently today, just wanted to let you know!" This helps you spot new or subtle problems before they become big, obvious ones that your rules would catch.

So, when you're building your own cool programs later on, you'll use alerting rules to catch things you expect to go wrong, and anomaly detection to find those sneaky, unexpected issues that might be hiding. This means you can keep your amazing digital projects running smoothly and catch problems early, just like a super gardener!

Alerting rules are your first line of defense in monitoring, acting as predefined conditions that, when met, trigger a notification. Think of them as 'if X happens, tell me'. These rules are typically based on metrics (like CPU usage, memory consumption, request latency, error rates) or specific log patterns (e.g., critical errors, failed authentications). For instance, a rule might be 'if CPU usage for service A is above 80% for 5 continuous minutes' or 'if the 99th percentile latency for API X exceeds 500ms for 10 minutes'. Setting effective alerting rules requires a good understanding of your system's normal behavior and operational thresholds, ensuring you're notified of real problems without being overwhelmed by noise.

Anomaly detection, on the other hand, takes a more sophisticated approach. Instead of relying on static thresholds, it uses algorithms to learn the 'normal' behavior patterns of your metrics and logs over time. It then identifies deviations from this learned baseline, flagging events that are statistically unusual. This is incredibly useful for catching subtle issues that might not breach a fixed threshold, or for systems with dynamic baselines (e.g., fluctuating traffic patterns). For example, a gradual increase in memory usage that stays below your 'critical' threshold but is still abnormal for that time of day, or a sudden, unexpected drop in successful requests during a normally busy period, could be caught by anomaly detection when a static rule might miss it.

Together, alerting rules and anomaly detection form a powerful duo for maintaining system health. Alerting rules are excellent for known, well-understood failure modes and clear-cut deviations, providing immediate notification for critical issues. Anomaly detection complements this by uncovering 'unknown unknowns' – problems that don't fit a predefined pattern, or subtle shifts in behavior that could indicate an impending larger issue. By combining both, you gain a comprehensive view, allowing you to proactively respond to both explicit critical failures and more nuanced, evolving operational challenges, ultimately reducing your Mean Time To Resolution (MTTR) and improving system reliability.

Key Takeaways

  • Alerting rules use predefined, static thresholds for known issues.
  • Anomaly detection identifies deviations from dynamically learned 'normal' behavior.
  • Rules catch explicit failures; anomaly detection finds subtle or evolving problems.
  • Both are crucial for proactive problem identification and reducing MTTR.
  • Combine them for comprehensive system health monitoring and fewer surprises.

Code Example

yaml
groups:
- name: backend-service-alerts
  rules:
  - alert: HighServiceErrorRate
    expr: sum(rate(http_requests_total{job="my-backend", status=~"5.."}[5m])) / sum(rate(http_requests_total{job="my-backend"}[5m])) * 100 > 5
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "High error rate on backend service {{ $labels.job }}"
      description: "The backend service {{ $labels.job }} has an error rate above 5% for 5 minutes. Current rate: {{ $value | humanize }}%"

How this code works

This YAML configuration defines an alerting rule designed to detect a high error rate on a backend service. Specifically, it watches for situations where more than 5% of all HTTP requests to my-backend are resulting in server-side 5xx errors, indicating a potential problem.

The alert rule, named HighServiceErrorRate, uses a expr (expression) written in PromQL to perform its calculation. This expr divides the rate of 5xx error requests by the rate of all requests for the job="my-backend" over the last five minutes, then multiplies by 100 to get a percentage. A subtle but important detail is the sum() function, which aggregates the rates across all instances of the backend service before calculating the overall percentage, ensuring a unified service-level error rate. The for: 5m instruction is key; it prevents the alert from firing immediately, requiring the high error condition to persist for five continuous minutes before becoming active. Finally, labels categorize the alert with severity: critical, and annotations provide helpful, dynamic information in the summary and description using templated values like the affected job and the exact error value.

Related Articles

Hand-picked reads to go deeper on logging, monitoring & observability.

Back to Backend Developer Roadmap