Quality monitoring for AI systems breaks into two distinct concerns that most teams conflate early on and then spend months separating. Operational metrics like p99 latency, HTTP error rates, and token counts behave like any other service metric: they are deterministic, fast to measure, and straightforward to alert on with a simple threshold. Quality metrics like answer faithfulness, retrieval precision, or tone adherence are probabilistic, expensive to evaluate, and require statistical baselines to alert on meaningfully. Treating them the same way leads to either missed regressions or a flood of noisy alerts that engineers learn to ignore.
The mental model that works: think of your dashboard as having three layers. The bottom layer is infrastructure, covering GPU/CPU utilization, queue depth, and memory. These are your "is the system alive" indicators. The middle layer is operational AI behavior: token usage per request, cost per user session, latency by model and prompt version, error rates by category. You already have tooling for this from the sibling lessons on metrics and logging. The top layer is output quality: faithfulness scores from an LLM judge, thumbs-up/thumbs-down rates from users, retrieval hit rates, and structured output parse failures. The top layer is the one that tells you whether the system is doing the right thing, not just running.
In a real-world production RAG system serving a customer support team, here is how a senior engineer would approach building this. First, they would instrument every LLM call to log the full trace: input, output, retrieved chunks, model, latency, and cost to a platform like Langfuse, Arize, or a custom Postgres table. Second, they would set up an async evaluation job that runs on a 5-10% sample of production traces using an LLM-as-judge (a pattern you covered in the evaluation lessons). This job scores faithfulness and relevance and writes those scores back to the same trace store. Third, they would build a Grafana dashboard with four panels: p50/p95 latency over time, cost per request over time, average faithfulness score (rolling 1h), and percentage of responses below the quality threshold. Fourth, they would wire PagerDuty or Slack alerts to fire when the rolling faithfulness average drops below a baseline by more than 1.5 standard deviations for 10 consecutive minutes. Not a hardcoded 0.6 threshold. A relative one.
The key tradeoff here is between coverage and cost. Evaluating 100% of production traffic with an LLM judge at GPT-4o prices is not realistic for most systems once you are past a few thousand requests per day. The typical approach is stratified sampling: always evaluate a random 5% baseline, plus 100% of any response that triggered a fallback, received a negative user rating, or exceeded a latency threshold. This gives you broad coverage and guaranteed coverage on the cases most likely to be bad. OpenAI or Anthropic for the judge model is fine for quality, but consider using a smaller judge like GPT-4o-mini for cost at scale and validating that it correlates well enough with the larger model on your specific task.
What changes at scale is mostly the granularity and the fan-out. At 10 users, you can afford a single Grafana dashboard refreshing every minute with a simple query. At 10,000 users, you need to segment by feature (chat vs. search vs. summarization), by user tier, by prompt version, and by geographic region because a problem in one segment is invisible in the aggregate. At 10 million users, you are pre-aggregating metrics in streaming pipelines before they reach your dashboard, and your alert system needs to account for seasonality and day-of-week patterns to avoid false positives during normal traffic spikes. Tools like Grafana Mimir, Prometheus with recording rules, and TimescaleDB with continuous aggregates handle this progression. The quality evaluation pipeline also needs to become fully asynchronous with a dedicated worker fleet rather than a cron job calling an API.
Alerts deserve their own discussion because most teams get them wrong in one of two directions: too sensitive (fires on every blip, engineers start ignoring it) or too loose (catches the incident after the user has already escalated). The fix is multi-window alerting. Set a fast alert with a tight threshold and a short window to catch catastrophic failures early. Set a slow alert with a looser threshold and a longer window to catch gradual drift. For example: alert immediately if error rate exceeds 20% over 2 minutes; alert if average faithfulness drops below 1.5 standard deviations from the 7-day mean over 30 minutes. Combine these with a dead-man's switch: if no quality evaluations have been processed in 2 hours, alert, because the evaluation pipeline itself may have died silently.
Key Takeaways
- Separate operational metrics from quality metrics; they require different alerting strategies.
- Use rolling baselines and statistical thresholds instead of hardcoded alert values.
- Sample LLM-as-judge evaluations in production to track quality without destroying your budget.
- Tune alert thresholds using historical data to avoid alert fatigue from day one.
Pro tips
- Store your baseline statistics (mean, stddev) in a database rather than hardcoding them, and recalculate them nightly against the last 7 days of data. A threshold that was correct at launch will drift out of calibration as usage patterns shift.
- Your LLM judge for production sampling should be a different model from the one you are monitoring. If GPT-4o is your production model and you use GPT-4o as the judge, a model-level regression and a judge drift can cancel each other out invisibly.
- Dead-man's switch alerts are more valuable than you think. If your evaluation pipeline silently dies at 3 AM, a threshold alert will never fire because no scores are being written. Always alert on the absence of expected events, not just the presence of bad ones.
- Segment your quality metrics by prompt version from day one, even before you have multiple versions. When you eventually change a prompt, you will want a clean comparison between the old and new cohorts, and retroactively tagging traces is painful.
Common pitfalls
- Mistake: Using a hardcoded absolute threshold like
score < 0.6for anomaly alerts. Fix: Compute a rolling baseline (7-day mean and stddev) and alert on z-score deviation so alerts adapt as your system improves. - Mistake: Evaluating 100% of production traffic with a large LLM judge at launch. Fix: Start with 5% stratified sampling and always evaluate negative feedback events; validate your judge correlates with human ratings first.
- Mistake: Putting all quality metrics into a single aggregate dashboard panel. Fix: Segment by feature, prompt version, and user tier so a regression in one area is not masked by stability in others.
- Mistake: Sending alerts directly to an individual's phone or email. Fix: Route to a team Slack channel with clear runbook links; on-call individuals burn out fast when alerts have no documented next step.
When to use which alerting strategy for AI quality metrics
| Option | Use when | Avoid when |
|---|---|---|
| Absolute threshold (score < 0.6) | You have a hard business requirement that a metric must never fall below a fixed value, e.g., safety classifier must flag >95% of harmful content. | Metrics are expected to improve over time or vary by traffic pattern; threshold will generate false positives or become stale. |
| Z-score / rolling baseline | Metric has natural variance and you want to detect anomalous drops relative to recent normal behavior, not an absolute floor. | You are in the first 48 hours of production and have no meaningful baseline yet; wait until you have at least a few hundred scored traces. |
| Percentage of requests below threshold | Average score looks fine but a tail of bad responses is growing; useful for catching p95-level degradation hidden by the mean. | Sample sizes are small (under 30 per window); percentage will be noisy and produce alert storms. |
| User feedback signal (thumbs down rate) | You have sufficient user volume to get statistically significant feedback within a reasonable alerting window (hours, not days). | Low-traffic systems where a single unhappy user can swing the rate 20 percentage points; lagging indicator anyway. |
Code Example
# langfuse>=2.0.0, requires LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY env vars
from langfuse import Langfuse
from datetime import datetime, timedelta
client = Langfuse()
# Pull faithfulness scores for the last 24 hours
traces = client.get_observations(
type="SCORE",
name="faithfulness",
from_timestamp=datetime.utcnow() - timedelta(hours=24),
)
scores = [t.value for t in traces if t.value is not None]
if scores:
avg = sum(scores) / len(scores)
low_count = sum(1 for s in scores if s < 0.6)
print(f"Avg faithfulness (24h): {avg:.3f}")
print(f"Responses below 0.6 threshold: {low_count}/{len(scores)}")
else:
print("No scores found in the last 24 hours")How this code works
This code monitors the quality of an AI system by analyzing its "faithfulness" scores over the last 24 hours. Its job is to provide a quick summary: the average faithfulness and how many responses fell below a predefined quality threshold, making it useful for a quality dashboard or anomaly detection.
The script starts by initializing the Langfuse client. It then uses client.get_observations to pull all entries classified as SCORE type with the name "faithfulness" from the last 24 hours, specifying this window with datetime.utcnow() - timedelta(hours=24). A subtle but important detail is the list comprehension scores = [t.value for t in traces if t.value is not None], which not only extracts the numerical score from each observation but also critically filters out any entries where the value might be None. This prevents errors in calculations if some scores were not recorded properly. Finally, if any scores are found, it computes their avg and counts how many are below 0.6, printing these insights. If no scores are retrieved, it gracefully reports that none were found.
Production-grade example
Retries on rate limits, structured logging, graceful Slack degradation, and z-score alerting over hardcoded thresholds.
# Production anomaly alerter: evaluates a sample of recent traces and fires Slack alert
# Deps: langfuse>=2.0.0, openai>=1.0.0, httpx>=0.25.0, tenacity>=8.0.0
import os
import logging
import statistics
import httpx
from datetime import datetime, timedelta
from tenacity import retry, stop_after_attempt, wait_exponential
from langfuse import Langfuse
from openai import OpenAI, RateLimitError, APITimeoutError
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger(__name__)
langfuse = Langfuse(
public_key=os.environ["LANGFUSE_PUBLIC_KEY"],
secret_key=os.environ["LANGFUSE_SECRET_KEY"],
)
oai = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
SLACK_WEBHOOK = os.environ["SLACK_ALERT_WEBHOOK"]
@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10),
retry=(RateLimitError, APITimeoutError))
def score_faithfulness(question: str, answer: str, context: str) -> float:
prompt = (
f"Rate how faithfully the answer is grounded in the context. "
f"Return only a float between 0.0 and 1.0.\n\n"
f"Context: {context[:800]}\nQuestion: {question}\nAnswer: {answer}"
)
resp = oai.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
max_tokens=10,
timeout=8,
)
raw = resp.choices[0].message.content.strip()
log.info("judge_response raw=%s input_tokens=%d", raw, resp.usage.prompt_tokens)
return float(raw)
def send_slack_alert(message: str) -> None:
try:
r = httpx.post(SLACK_WEBHOOK, json={"text": message}, timeout=5)
r.raise_for_status()
except Exception as exc:
log.error("slack_alert_failed error=%s", exc) # degrade gracefully
def run_quality_check(lookback_hours: int = 1, sample_rate: float = 0.05,
z_score_threshold: float = 1.5) -> None:
since = datetime.utcnow() - timedelta(hours=lookback_hours)
traces = langfuse.get_observations(type="GENERATION", from_timestamp=since, limit=500)
sample = [t for i, t in enumerate(traces) if i % int(1 / sample_rate) == 0]
log.info("quality_check total=%d sampled=%d", len(traces), len(sample))
if len(sample) < 5:
log.warning("insufficient_sample n=%d skipping alert evaluation", len(sample))
return
scores = []
for obs in sample:
try:
meta = obs.metadata or {}
score = score_faithfulness(
question=meta.get("user_query", ""),
answer=obs.output or "",
context=meta.get("retrieved_context", ""),
)
scores.append(score)
langfuse.score(trace_id=obs.trace_id, name="faithfulness_prod", value=score)
except (ValueError, KeyError) as exc:
log.warning("score_parse_error trace=%s err=%s", obs.trace_id, exc)
if not scores:
log.error("no_valid_scores firing dead-man alert")
send_slack_alert(":warning: Quality eval produced zero valid scores. Check eval pipeline.")
return
avg = statistics.mean(scores)
baseline_avg, baseline_std = 0.82, 0.08 # replace with DB lookup in real system
z = (avg - baseline_avg) / baseline_std if baseline_std else 0
log.info("quality_result avg=%.3f baseline=%.3f z=%.2f n=%d", avg, baseline_avg, z, len(scores))
if z < -z_score_threshold:
pct_below = sum(1 for s in scores if s < 0.6) / len(scores) * 100
send_slack_alert(
f":rotating_light: Faithfulness regression detected\n"
f"Current avg: {avg:.3f} (baseline: {baseline_avg:.3f}, z={z:.2f})\n"
f"{pct_below:.0f}% of sampled responses scored below 0.6\n"
f"Sample size: {len(scores)} traces from last {lookback_hours}h"
)
if __name__ == "__main__":
run_quality_check()How this code works
This Python script acts as an automated quality monitoring and alerting system for AI applications. Its job is to periodically check how well an AI assistant's responses are grounded in its provided context (known as "faithfulness") and to notify developers via Slack if this quality drops unexpectedly. This helps maintain AI system reliability and provides early warnings about potential issues, fitting into AI Observability & Monitoring.
The script starts by connecting to Langfuse to fetch recent AI GENERATION traces and OpenAI to power an AI judge. The score_faithfulness function is key; it uses gpt-4o-mini to evaluate a response, rating it from 0.0 to 1.0. This function is resilient, thanks to @retry, which automatically retries RateLimitError or APITimeoutError. The main run_quality_check function then samples recent AI interactions, evaluates their faithfulness, and calculates the avg score. It compares this average to baseline_avg and baseline_std values to compute a z score. If the z score falls below z_score_threshold, indicating a significant quality drop, a send_slack_alert is triggered. A subtle but important detail for beginners is that the baseline_avg and baseline_std are hardcoded as placeholders; in a production system, these would typically be dynamically looked up from a database storing historical quality metrics for robust anomaly detection.
Practice & master
Try the exercise, check your understanding, then mark this lesson mastered to track your path to pro.
Exercise
Build a quality report script that reads the last 2 hours of LLM call logs from a local JSON file, calculates average faithfulness and the percentage of responses below 0.65, and prints a warning to the console if the percentage exceeds 20%. Use the log schema: {user_query, answer, retrieved_context, faithfulness_score}.
import json
import sys
from pathlib import Path
LOG_FILE = Path("llm_logs.jsonl") # one JSON object per line
THRESHOLD = 0.65
WARN_PERCENT = 20.0
def load_logs(path: Path) -> list[dict]:
# TODO: read the JSONL file and return a list of dicts
pass
def compute_report(logs: list[dict]) -> dict:
# TODO: extract faithfulness_score from each log entry
# TODO: compute average score and percentage below THRESHOLD
# Return dict with keys: avg_score, pct_below, total
pass
def print_report(report: dict) -> None:
# TODO: print avg_score and pct_below
# TODO: if pct_below > WARN_PERCENT, print a warning line
pass
if __name__ == "__main__":
logs = load_logs(LOG_FILE)
if not logs:
print("No logs found.")
sys.exit(0)
report = compute_report(logs)
print_report(report)Quick check
Your faithfulness scores average 0.81 today, down from 0.84 yesterday. Should you page on-call?
Why is evaluating 100% of production traces with GPT-4o as your quality judge often a bad idea at scale?
What does a dead-man's switch alert catch that a standard threshold alert misses?