# How to Pick LLM Application Observability Metrics [2026]

> Token logs are table stakes. Here’s the minimum set of LLM application observability metrics for tools, RAG, caching, refusals, and privacy-safe logging that actually debugs production incidents.

- Canonical: https://www.kunalganglani.com/blog/llm-observability-metrics
- Author: Kunal Ganglani
- Published: 2026-08-15 · Updated: 2026-08-15
- Category: Cloud and DevOps · Tags: llmops, observability, production-ai, opentelemetry, rag

## TL;DR

Token logs won’t tell you why an AI feature got slow, started failing, or began giving worse answers. You need a small set of signals that cover the whole request: the model call, document retrieval, tool calls, retries, caching, and safety outcomes. This guide gives you a minimum checklist you can copy into a dashboard, plus a simple way to organize everything as traces, metrics, and events so it plugs into your existing monitoring stack. The key idea is to log structure, not private text. That keeps debugging fast without turning your telemetry into a data leak.

Last year I watched a team burn three days on an “LLM incident” that had nothing to do with the model. p99 doubled, costs spiked, quality dipped. The only thing they had was token counts and a big pile of prompt/response logs. So the debugging session turned into archaeology.

You’ll finish this in about 45 minutes with a working “minimum viable” LLM observability setup: a trace model (LLM + retrieval + tool calls + retries), a short metrics list, and a privacy-safe logging policy you can ship without storing raw prompts.

If you’re only collecting token counts and prompt/response logs, you don’t have **LLM application observability metrics**. You have an expensive transcript.

This post is about the signals that explain the stuff that actually breaks in production: tool calls that time out, retrieval that quietly returns garbage, caches that “help” by serving wrong answers faster, and safety filters that suddenly start refusing half your traffic.

## What is LLM application observability?

LLM application observability is instrumenting an LLM-powered system with traces, metrics, and logs so you can explain latency, cost, failures, and answer-quality changes across the full request path. That means the model call, retrieval, tool/function calls, retries, caching, and safety.

![Nvidia logo on a green background with abstract spheres.](https://cdn.sanity.io/images/vzekdneq/production/0a35f113b2a927c93b894c441a8e48d93a53f915-1200x675.webp)

The important part is “full request path”. The model is rarely the product.

I learned this the hard way building the Walmart conversational commerce chatbot at Firework (Zealsight). We handled **millions of queries daily** with **sub-second responses**, and the real wins came from tracing the pipeline (retrieval + streaming + orchestration), not staring at token logs. At that scale, “the LLM was slow” is basically never a root cause. It’s a symptom you slap on anything you can’t see.

## The minimum signals checklist (copy this into your dashboard)

This is the smallest set of signals I’d accept before calling an LLM feature “production-ready”. Keep it tight. You can always add the fancy stuff later. What you can’t do is debug p99 with vibes.

![Nvidia logo on a green background with abstract 3D elements.](https://cdn.sanity.io/images/vzekdneq/production/d98eddd662a70d154e51aa8c5f6e564dd1f76b07-1200x675.webp)

| Signal | Why it matters | How to capture | Suggested metric names |
| --- | --- | --- | --- |
| End-to-end request latency (p50/p95/p99) | Users feel p95. Incidents live in p99. | Server span around the whole request. | `llm_app.request.latency_ms` |
| LLM call latency + TTFT | Separates model slowness from your pipeline slowness. | Span per model call; record TTFT as event/exemplar. | `llm.call.latency_ms`, `llm.call.ttft_ms` |
| Tool/function call success + latency | Tools fail more than models. Also where most hidden retries happen. | Span per tool call with status + attempts. | `tool.call.success_rate`, `tool.call.latency_ms`, `tool.call.retries_total` |
| Schema/validation mismatch rate | “It returned JSON” is not correctness. | Validate tool args and structured outputs. | `tool.call.schema_mismatch_rate` |
| Retrieval latency breakdown | RAG regressions often start as latency regressions. | Separate spans: embed → search → rerank. | `rag.embed.latency_ms`, `rag.search.latency_ms`, `rag.rerank.latency_ms` |
| Empty retrieval rate | “No docs found” should be loud, not silent. | Count retrievals with 0 usable chunks. | `rag.empty_result_rate` |
| Top-k actually used | If you ask for k=20 but use 3, you’re paying for nothing. | Log `k_requested`, `k_returned`, `k_used`. | `rag.k_used` (histogram) |
| Chunk duplication/overlap | High overlap inflates tokens and reduces diversity. | Hash chunk IDs; compute overlap ratio. | `rag.chunk_overlap_ratio` |
| Cache hit rate (prompt/embedding/semantic) | Cache changes cost and latency by multiples, and can regress correctness. | Count hits/misses by cache tier + version. | `cache.prompt.hit_rate`, `cache.embedding.hit_rate`, `cache.semantic.hit_rate` |
| Refusal / safe-completion rate | Safety behavior changes over time and by cohort. | Classify outcome category without storing raw prompt. | `safety.refusal_rate`, `safety.safe_completion_rate` |
| PII redaction rate | Proves your privacy controls are working. | Redact before export; count redactions. | `privacy.redaction_events_total` |
| Sampling + retention compliance | Prevents “observability becomes a data leak”. | Enforce TTL + sampled logging. | `telemetry.sample_rate`, `telemetry.retention_days` |

A note on numbers: graph p50/p95/p99 for anything latency-related. Keep at least **7 days** of metrics at full fidelity. Logs can be sampled way harder (often **0.1%–1%** for high-volume apps) if you have traces and exemplars.

## How to structure LLM telemetry as traces, metrics, and events (so your APM works)

Most teams accidentally create a new observability universe for LLMs. They ship a vendor dashboard, add a giant JSON log blob, sprinkle some notebooks around. Six months later, nobody trusts any of it, and on-call falls back to… reading prompts.

![Abstract digital scene with nvidia logo and geometric shapes.](https://cdn.sanity.io/images/vzekdneq/production/2860da211d80fbca021609ef6b41bae07618ed75-1200x675.webp)

I’m blunt about this: **force LLM telemetry into standard primitives**.

- **Traces** answer: “Where did the time go?”
- **Metrics** answer: “Is it getting worse over time, and for whom?”
- **Logs/events** answer: “What exactly happened on this one weird request?”
If you already run OpenTelemetry (OTel), you’re most of the way there. I wrote a deeper implementation guide in [OpenTelemetry Instrumentation for AI Agents [2026]: Ship It](/blog/opentelemetry-ai-agents-instrumentation). I’m not going to duplicate the schema here. What you need from this post is a trace shape you can standardize on and the minimum set of attributes that won’t get you hauled into a privacy review.

### The trace shape I use for LLM apps

You want one trace per user request, with spans that map to real pipeline boundaries. Not “AI stuff happened here”. Real boundaries.

1. **`http.request`** (root span)
1. **`llm.plan`** (optional, if you do multi-step / agentic planning)
1. **`rag.retrieve`** (optional, if you do [RAG](/glossary/rag))
  - `rag.embed_query`
  - `rag.vector_search` (or hybrid search)
  - `rag.rerank`
1. **`tool.call.<name>`** (0..N)
1. **`llm.generate`** (the final response)
1. **`response.stream`** (optional; track streaming UX separately)
The rule I won’t compromise on: **every external dependency gets its own span**.

Payments API. Span.

Postgres lookup. Span.

Vector DB call. Span.

If you don’t do this, your p99 chart is just a mystery novel where every suspect has an alibi.

Treat the LLM like a microservice. Because that’s what it is in your architecture.

### Minimum span attributes that don’t leak data

I try to avoid raw prompt logging by default. Instead, I put join keys on spans so I can correlate behavior across versions and cohorts.

- `request_id` (server-generated)
- `user_cohort` (coarse: `free`, `pro`, `internal`)
- `model.name` + `model.version` (or provider + deployment)
- `prompt_template_id` + `prompt_version`
- `rag.corpus_id` + `rag.index_version`
- `tool.name` + `tool.version`
- `cache.tier` + `cache.key_version`
- `safety.outcome` (enum)
Those version fields matter more than people think. Half of the “LLM randomness” incidents I see are actually “we changed an index / prompt / tool schema and forgot to correlate it”. Then everyone argues about temperature settings like that’s the problem.

If you want a concrete example of versioning and pipeline identity, my [AI Engineering Evals: Regression Gates for Prompts, Tools, RAG [2026]](/blog/ai-engineering-evals-gates) post shows how I tie observability back to eval gates.

## How to trace and measure tool/function calls (latency, errors, retries, schema mismatch)

Tool calls are where demos go to die.

Your LLM can be perfect and your app still fails because:

- the tool timed out at **2,000 ms**
- the LLM produced invalid args **3 times** and you retried
- the upstream API returned **429** and you backoff-looped
- the tool “succeeded” but returned an empty payload (a silent failure)
### What I measure for each tool

For each tool/function, I want five things:

- **Success rate**: `success / attempts`
- **Latency**: p50/p95/p99 of the tool span
- **Retry count distribution** (histogram, not a single average)
- **Error categories** (timeouts vs 4xx vs 5xx vs validation)
- **Schema mismatch rate** (LLM produced args that didn’t validate)
If you do only one thing: **break retries out as their own spans**.

Retries are the sneakiest source of “why did p99 double?” because the user only sees the final response. You quietly did 2–4 extra round trips and congratulated yourself for “recovering”.

### How to attribute downstream failures to the right span

I see a lot of traces where everything is marked “OK” except the root span. That’s useless.

A simple rule:

- If the tool request failed (timeout / non-2xx), mark the tool span as error.
- If the tool returned data but you rejected it (schema/validation), mark the tool span as error.
- If the tool succeeded but the LLM refused to use it, do not mark the tool span as error. Mark a **decision event** on the `llm.plan` or `llm.generate` span.
This distinction is how you stop blaming infra for product behavior.

If you’re building [AI agents](/pillars/ai-agents) or doing agent orchestration, tool-call observability is your real “agent reliability” layer. The model is the easy part.

## Cache metrics that actually matter (and how to detect cache-caused regressions)

Caches are a first-class dependency in LLM apps. Pretending they’re just an implementation detail is how you get blindsided.

You usually have at least two of these. Most teams end up with all three:

1. **Prompt cache** (exact match, or templated)
1. **Embedding cache** (query embedding reuse)
1. **Semantic cache** (approximate match, “close enough”)
If you don’t instrument caches, you’ll misread both cost and quality.

A semantic cache can cut cost by **30%–60%** in high-repeat workloads. It can also destroy correctness when the query is similar but not identical. Faster wrong answers are still wrong.

I go deeper on cost levers in [Reduce LLM API Costs 60%: 6 Techniques [2026]](/blog/reduce-llm-api-costs-production) and per-task accounting in [Agent Per-Task Cost Calculation [2026]: Retries, Tools, Caching](/blog/agent-per-task-cost-calculation). Here I’m focusing on what to observe so you can catch regressions before support tickets do.

### Minimum cache metrics per tier

- `cache.<tier>.hit_rate`
- `cache.<tier>.lookup_latency_ms` (p95 matters)
- `cache.<tier>.stale_served_rate`
- `cache.<tier>.bypass_rate` (how often your app skips cache)
- `cache.<tier>.key_version_mismatch_rate`
That last one is your regression smoke alarm.

If you change prompt templates or tool schemas and don’t bump cache key versions, your cache becomes a correctness bug factory. You’ll see “great latency, bad answers” and the Slack thread will immediately blame the model.

### Detecting cache-caused quality regressions without storing prompts

You can detect most cache regressions with three join keys:

- `prompt_version`
- `cache.key_version`
- `eval_outcome_bucket` (or a proxy)
If you run continuous evals, correlate “bad outcome” with cache hit/miss.

If you don’t have full evals yet, use proxies:

- user rephrase rate within **30 seconds**
- user click-through on citations (for RAG)
- “thumbs down” rate
- session abandonment after answer
They’re imperfect. They beat guessing.

## RAG observability metrics for production (and how to debug answer quality drops)

Retrieval-Augmented Generation (RAG) is where most “LLM apps” become real products. It’s also where people ship the least observability.

RAG failures are usually boring:

- retrieval returned nothing
- retrieval returned duplicates
- retrieval returned irrelevant chunks
- reranker got slower and you silently lowered k
- index got rebuilt and semantics shifted
On the Walmart chatbot, one lesson kept repeating: **retrieval quality dominated answer quality at scale**. Model upgrades were incremental. Retrieval regressions were catastrophic.

### Minimum RAG observability metrics

- **Latency** by stage: embedding, search, rerank
- **Empty-result rate** (`k_returned = 0` or `usable_chunks = 0`)
- **Top-k distribution**: requested vs returned vs used
- **Chunk overlap ratio**: duplicates / near-duplicates in the final context
- **Citation coverage**: how often your answer cites at least 1 retrieved chunk
- **Context token budget**: tokens spent on retrieved context vs generation
If you want my take on why “just increase the context window” is a trap, see [RAG Context Window Limits: Why Bigger Is Not Better [2026]](/blog/rag-context-window-limitations).

### Debugging playbook: “answer quality dropped”

When someone tells me “quality dropped”, I don’t start by debating model upgrades. I run this sequence:

1. Check `rag.empty_result_rate` for a spike. If it goes from **0.5%** to **5%**, that’s your incident.
1. Check `rag.search.latency_ms p99`. If p99 jumped, some callers will time out and fall back.
1. Compare `rag.index_version`. If the index rebuilt, correlate the quality change with the version boundary.
1. Look at `rag.chunk_overlap_ratio`. If overlap jumped, your chunking pipeline probably changed.
1. Check `rag.k_used`. If you silently started using fewer chunks, you probably introduced a budget clamp.
This is exactly why you track index identity in spans. Without it, you can’t correlate, so you can’t fix.

If you’re doing retrieval-augmented generation seriously, you’ll eventually want offline evals and replay harnesses. My [Agent Evaluation Harness [2026]: Replay, Rubrics, CI Gates](/blog/agent-evaluation-harness-replay) post is the next step.

## Refusal rate and safety outcomes (without storing sensitive prompts)

If your refusal rate is “unknown”, you don’t have a production system. You have a liability.

Track these as metrics:

- `safety.refusal_rate`
- `safety.safe_completion_rate`
- `safety.policy_trigger_rate` (by category)
Then correlate them with:

- `model.version` (providers change behavior)
- `prompt_version` (you change behavior)
- `tool.name` (tools can trip policy boundaries)
- user cohort
A refusal spike can come from:

- an upstream model update
- an overly aggressive content filter
- a new tool that surfaces sensitive data
- a [prompt injection](/blog/prompt-injection-2026-owasp-llm-vulnerability) chain that tries to jailbreak the system
If you’re not threat-modeling this, start here: [AI security](/blog/ai-security-complete-guide). I also keep a checklist in AI security that matches how I think about “safe by default” telemetry.

### How to classify outcomes without logging prompts

Use an enum outcome plus a coarse reason code.

Example outcomes:

- `ok`
- `refused`
- `partial`
- `safe_completion`
- `tool_blocked`
Reason codes can be your taxonomy. Keep it stable and versioned. If you change your categories every month, you’re destroying your own trend lines.

The point is to measure behavior without hoarding content.

## Privacy-safe LLM logging: what to store, what to hash, sampling, retention

“Just log the prompts” is the GenAI equivalent of “just run it as root”. It works right up until the day it ruins your week.

I’m not saying “never store prompts”. I’m saying: **make prompt storage an explicit, reviewed exception**.

I wrote a dedicated privacy playbook in [Data Privacy in RAG Redaction and Retention [2026 Playbook]](/blog/data-privacy-rag-redaction-retention) and the security angle in [Prevent Sensitive Data Leakage in RAG: The 2026 Playbook](/blog/prevent-sensitive-data-leakage-rag). Here’s the minimum policy I’d ship.

### Default: store structure, not content

Store:

- prompt template ID + version
- tool names + versions
- retrieval corpus ID + index version
- token counts and cost estimates
- safety outcomes
- hashes of prompts/responses (salted)
Don’t store by default:

- raw user prompts
- raw model outputs
- raw retrieved chunks
If you need debugging content, use short-lived secure capture:

- gated behind a feature flag
- enabled for a single `request_id`
- TTL of **24 hours** (or less)
- access logged and limited to on-call
### Redaction and hashing that’s actually useful

- Redact obvious PII before telemetry export.
- Hash the canonicalized prompt (lowercased, whitespace-normalized) with a rotating salt.
Canonicalize because otherwise you can’t correlate duplicates.

Rotate salts because otherwise your logs become a tracking vector.

### Sampling strategy for high-volume LLM apps

A sane default I’ve used:

- **100% metrics** (cheap)
- **10% traces** for baseline, plus **100% traces on errors**
- **0.1% logs** (structured events), plus “burst” logging for a single incident window
If you’re doing millions of requests/day, logging every prompt is not just risky. It’s expensive.

If you need to reason about cost tradeoffs, I keep my own datasets on pricing and performance. Based on the benchmark and cost data I maintain at [kunalganglani.com/llm-prices](/llm-prices) and [kunalganglani.com/llm-benchmarks](/llm-benchmarks), the gap between “fast cheap model” and “frontier model” can be multiple times in $/request. Observability that can attribute cost to spans is how you stop arguing in Slack and start making decisions.

## Implementation notes: don’t create an observability silo

The goal isn’t “an LLM dashboard”. The goal is: when p99 spikes, your on-call uses the same workflows they use for every other service.

A few practical moves:

- Emit tool spans as normal client spans. Your APM already knows how to graph dependency latency.
- Put LLM metadata on spans as attributes, not log blobs.
- Use exemplars to attach “this trace is a p99 outlier” to your histograms.
- Keep a strict boundary between telemetry and product analytics. They can share IDs, not raw content.
If you’re running production AI systems, treat observability changes like API changes. Version them. Review them. Test them.

## The uncomfortable prediction

Within 12 months, “token logs” will be viewed the way we now view `console.log` in a backend service. Fine for a demo. Embarrassing in production.

If you’re building LLM apps, pick **10–12 signals**, wire them into traces/metrics/events, and ship a privacy policy with the same seriousness you ship auth.

Then do the hard part. When your dashboards light up, don’t blame the model. Fix your pipeline.

Photo by 1981 Digital on Unsplash.