# RAG Evaluation Metrics for Retrieval Quality: My Production Playbook

> If your RAG app got worse after an embeddings or chunking change, grading answers won’t tell you why. Here’s a retrieval-first eval workflow: recall@k, MRR, citation accuracy, leakage tests, and a frozen offline corpus so regressions are real—not vibes.

- Canonical: https://www.kunalganglani.com/blog/rag-evaluation-metrics-retrieval-quality
- Author: Kunal Ganglani
- Published: 2026-09-14 · Updated: 2026-09-14
- Category: AI and Machine Learning · Tags: rag, evals, llmops, retrieval, production-ai

## TL;DR

RAG systems fail more often in the “find the right info” step than in the “write a good answer” step. If you only grade final answers, you can’t tell whether a regression came from embeddings, chunking, filters, or a reranker change. This guide shows how to build a repeatable evaluation setup: freeze an offline document corpus, label queries to the right passages, and track retrieval metrics like recall@k and MRR. It also adds practical tests for citation accuracy and chunk leakage so your scores don’t lie. The payoff is simple: faster debugging and fewer surprises in production.

## RAG Evaluation Metrics for Retrieval Quality: My Production Playbook

The fastest way to waste a week on RAG is to “evaluate” it by grading final answers.

![A digital dashboard displaying marketing metrics including CTR and quality score on a screen](https://cdn.sanity.io/images/vzekdneq/production/4a03eb62590ae9a325dad33276a520c3428eb50b-1200x675.webp)

You tweak chunking. Or switch embeddings. Or tighten a metadata filter. Suddenly the app feels worse. Your LLM-as-judge score drops. Everyone argues about prompt wording. Nobody can explain *what actually broke*.

This post is about **RAG evaluation metrics retrieval quality**. Not “LLM-as-judge vibes.” The retrieval layer is where most real failures hide.

I’m blunt about this because I’ve watched perfectly respectable demo evals ship straight into production pain. When we built the Walmart conversational commerce chatbot (Firework/Zealsight, 2022–2024), it handled **millions of queries daily** at **sub-second** latency and drove a **400% product engagement lift**. The thing that kept deciding whether users got good answers was not a fancier model. It was whether we reliably fetched the right evidence.

## What is RAG evaluation metrics retrieval quality?

**RAG evaluation metrics retrieval quality are measurements that tell you whether your retriever (and reranker) is fetching the right passages for a query, independent of how smooth the final LLM answer sounds.**

![A digital dashboard displaying marketing metrics including CTR and quality score on a screen](https://cdn.sanity.io/images/vzekdneq/production/4a03eb62590ae9a325dad33276a520c3428eb50b-1200x675.webp)

A lot of teams only score the final response with an LLM judge. That’s not useless. It’s just a lousy diagnostic tool. If embeddings drift, your chunker changes, your index params get “optimized,” or a filter starts returning empty sets, answer grading will only tell you “things got worse” and then shrug.

Pinecone says the quiet part out loud: your RAG pipeline is bounded by retrieval. They put it as, “Your RAG pipeline is only as performant as your retrieval phase is accurate,” and recommend monitoring retrieval with precision/recall-style metrics on the top results returned ([Pinecone](https://www.pinecone.io/learn/series/rag/rag-evaluation/)). That’s the right mental model.

The tooling world is finally catching up too. LangSmith’s evaluation docs (modified **2026-09-11**) push you to split systems into components and curate ground-truth sets per component ([LangSmith team](https://docs.smith.langchain.com/evaluation)). That “component thinking” is what most RAG teams skip when they treat eval as “judge the answer and ship.”

Here’s the workflow I use:

1. Freeze an offline corpus (docs + chunker settings + embedding model + index params).
1. Build a labeled dataset: queries → relevant passage IDs.
1. Compute retrieval metrics (recall@k, precision@k, hit rate, MRR; optionally nDCG).
1. Score citation accuracy as a first-class metric.
1. Run leakage/contamination tests so you don’t fool yourself.
1. Separate retriever vs reranker metrics.
1. Set “good enough” thresholds and production alerts.
## RAG evaluation measures your pipeline’s performance (but debug it in layers)

A RAG pipeline is a chain: query → retrieval → prompt construction → generation. If you score it like a blob, you’ll “fix” the wrong thing.

![A digital dashboard displaying marketing metrics including CTR and quality score on a screen](https://cdn.sanity.io/images/vzekdneq/production/4a03eb62590ae9a325dad33276a520c3428eb50b-1200x675.webp)

Arize frames this as two tracks: retrieval evaluation and response evaluation. In their workshop write-up, **Shittu Olumide** describes them as distinct workflows because retrieval and generation failures require different fixes ([Shittu Olumide](https://arize.com/blog/evaluate-rag/)). This maps cleanly to reality. When teams conflate the two, they end up prompt-tuning to hide a retrieval bug. That’s not engineering. That’s makeup.

Concrete example:

- You switch embeddings from `text-embedding-3-large` to something cheaper.
- Your answer quality drops 8% (as judged by an LLM).
- The root cause might be **recall@10** falling from **0.86 → 0.71** on your golden set, not the model “getting dumber.”
Answer-only evals won’t tell you that. Retrieval evals will.

If you want a broader framework for how this plugs into production workflows, pair this post with my [AI in production](/pillars/ai-engineering-production) pillar and the more general [AI agents](/pillars/ai-agents) work. Retrieval is the same kind of unsexy infrastructure problem as agent orchestration. If you can’t observe it, you can’t ship it.

## Understanding binary relevance metrics (and how to compute them)

Binary relevance is the simplest setup: each retrieved passage is relevant or it isn’t.

People love to sneer at “binary” anything. I don’t. For most production RAG systems, binary metrics are the highest ROI thing you can do this month.

### The metric cheat sheet (use this as your dashboard backbone)

| Metric | What it measures | Good for | Pitfall |
| --- | --- | --- | --- |
| **Recall@k** | Of all relevant passages, what fraction appeared in top `k`? | “Did we fetch the needed info?” | Needs labeled relevant IDs; doesn’t care about rank within top `k`. |
| **Precision@k** | Of top `k`, what fraction is relevant? | “Are we stuffing garbage context?” | High precision can hide low recall if your `k` is small. |
| **Hit rate@k** | Did we retrieve *at least one* relevant passage in top `k`? | Customer-support style Q&A where one good doc is enough | Can look good even when coverage is bad. |
| **MRR** (Mean Reciprocal Rank) | How early the first relevant passage appears | When rank matters (small context window, rerankers) | Only considers the first relevant result. |
| **nDCG@k** | Ranking quality with graded relevance | When “partially relevant” exists | Harder labeling; easier to overfit. |
| **Context precision/recall** (RAGAS) | Relevance/noise in retrieved context | RAG-specific evaluation | Depends on judging scheme; can blur retrieval vs generation. |
| **Citation accuracy** | Are cited passages actually supporting claims? | Grounding and trust | Needs a citation protocol, not vibes. |

RAGAS explicitly splits context-focused metrics like Context Precision / Context Recall from response-focused metrics like Faithfulness ([Ragas team](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/)). I like that taxonomy even if you never touch RAGAS.

### How to compute recall@k, precision@k, hit rate, and MRR on a labeled dataset

You need a dataset shaped like this:

- Query `q`
- Set of relevant passage IDs `R(q)` (1…N)
- Retrieved ranked list `L_k(q)` for top `k` passage IDs
Definitions:

- **Recall@k(q)** = `|R(q) ∩ L_k(q)| / |R(q)|`
- **Precision@k(q)** = `|R(q) ∩ L_k(q)| / k`
- **HitRate@k(q)** = `1` if `R(q) ∩ L_k(q)` non-empty else `0`
- **RR(q)** (reciprocal rank) = `1 / rank(first relevant in L)` else `0`
- **MRR** = average of `RR(q)` across queries
If you have multiple relevant passages, MRR still uses the first hit. That’s not a bug. It answers a specific question: “How quickly do we surface *something* relevant?”

Where teams get sloppy is labeling multiple relevant chunks. My rule of thumb:

- If the query is “what is the refund policy for X”, label **all passages** that contain the authoritative policy text.
- If the query is “is product A compatible with product B”, label the passages that actually mention the relationship.
You’ll notice I’m using *passage IDs*, not raw text. That’s deliberate. If you anchor your labels to text, a minor chunking tweak turns your eval set into confetti.

Here’s a minimal, runnable Python script that computes these metrics from a JSONL file.

```python
import json
from statistics import mean

def recall_at_k(relevant, retrieved_k):
    relevant = set(relevant)
    if not relevant:
        return 0.0
    return len(relevant.intersection(retrieved_k)) / len(relevant)

def precision_at_k(relevant, retrieved_k):
    relevant = set(relevant)
    if not retrieved_k:
        return 0.0
    return len(relevant.intersection(retrieved_k)) / len(retrieved_k)

def hit_rate_at_k(relevant, retrieved_k):
    relevant = set(relevant)
    return 1.0 if relevant.intersection(retrieved_k) else 0.0

def reciprocal_rank(relevant, retrieved_ranked):
    relevant = set(relevant)
    for idx, pid in enumerate(retrieved_ranked, start=1):
        if pid in relevant:
            return 1.0 / idx
    return 0.0

def load_jsonl(path):
    with open(path, "r", encoding="utf-8") as f:
        for line in f:
            if line.strip():
                yield json.loads(line)

def eval_retrieval(path, k=10):
    recalls = []
    precisions = []
    hits = []
    rrs = []

    for row in load_jsonl(path):
        relevant = row["relevant_passage_ids"]  # list[str]
        retrieved = row["retrieved_passage_ids"]  # ranked list[str]

        retrieved_k = retrieved[:k]

        recalls.append(recall_at_k(relevant, retrieved_k))
        precisions.append(precision_at_k(relevant, retrieved_k))
        hits.append(hit_rate_at_k(relevant, retrieved_k))
        rrs.append(reciprocal_rank(relevant, retrieved))

    return {
        f"recall@{k}": mean(recalls) if recalls else 0.0,
        f"precision@{k}": mean(precisions) if precisions else 0.0,
        f"hit_rate@{k}": mean(hits) if hits else 0.0,
        "mrr": mean(rrs) if rrs else 0.0,
        "n_queries": len(recalls),
    }

if __name__ == "__main__":
    import argparse

    parser = argparse.ArgumentParser()
    parser.add_argument("--path", required=True)
    parser.add_argument("--k", type=int, default=10)
    args = parser.parse_args()

    print(eval_retrieval(args.path, k=args.k))
```

A sane starting point is **50–200 queries** in your golden set. For a brand new system, I’ll start with **30** just to get the harness working, then grow it weekly.

LangSmith recommends curating “**5–10 examples** of what good looks like for each critical component,” including good retrievals and good answers ([LangSmith team](https://docs.smith.langchain.com/evaluation)). That’s a great minimum. For production, you’ll want more than 10 pretty quickly because retrieval failures are spiky and weirdly domain-specific.

## Understanding graded relevance metrics (when nDCG beats MRR)

Binary relevance treats “somewhat relevant” and “dead-on” as the same thing. In a lot of enterprise RAG, that’s simply not true.

If you’re retrieving:

- a policy page (high relevance)
- a Slack thread quoting the policy (medium relevance)
- a random customer complaint mentioning the policy (low relevance)
…you probably want the policy page at rank 1, every time.

That’s where **nDCG@k** is worth the labeling pain. You assign a relevance grade per passage (0/1/2/3), discount by rank (log-based), and normalize by the ideal ordering.

When I reach for graded metrics:

- You have multiple levels of correctness.
- Your context window is tight, so rank is survival.
- You’re evaluating rerankers that shuffle near-ties.
When I don’t bother:

- Your domain is mostly factual lookup where relevant is relevant.
- You’re early and still fixing chunking and filters.
If your team is already using LlamaIndex evaluators for answer/context relevancy, they explicitly separate scoring for answers vs retrieved contexts and return a 0–1 score (LlamaIndex team). That separation is good. I still prefer IR-style metrics as bedrock because they don’t depend on a judge model’s mood.

## Retrieval evaluation: build a frozen offline corpus (so scores mean something)

Most “offline evals” aren’t offline. They’re “run the tests against whatever happens to be in the index today.”

That makes your metrics non-comparable across releases. You think you improved retrieval, but you actually just re-ingested docs. Or changed the chunker. Or both.

A **repeatable offline RAG evaluation dataset** needs an immutable corpus and explicit versioning.

Here’s the recipe.

### Step 1: Freeze documents with provenance

Store:

- `doc_id`
- `source_uri` (URL, S3 path, Confluence page ID)
- `source_etag` or content hash
- `retrieved_at` timestamp
- `content` (raw)
If you can’t freeze raw content (compliance), freeze a hashed representation plus a stable export process.

### Step 2: Freeze chunking

Chunking is where evals get accidentally gamed.

Store the chunking config:

- `chunker_name` and version (e.g., `recursive_character@1.3`)
- `chunk_size` (tokens or chars)
- `overlap` (tokens)
- preprocessing rules (strip boilerplate? remove nav?)
If you change any of these, you’re not “re-running eval.” You’re evaluating a *new retriever*.

I’ve seen teams bump overlap from **50 → 200 tokens** and celebrate a jump in answer quality. Sometimes that’s real. Other times you just made it easier for a chunk to carry the answer around like a cheat sheet.

### Step 3: Freeze embeddings and index params

Store:

- embedding model name + version
- vector dimension (e.g., **1536**)
- normalization (cosine vs dot)
- ANN index algorithm + params (HNSW `M`, `efConstruction`, query `efSearch`)
If you don’t freeze ANN params, your “retrieval regression” might just be `efSearch` drifting from **64 → 16** during a cost-cutting sprint.

### Step 4: Assign stable passage IDs

Every chunk gets a `passage_id` like:

`{doc_id}:{chunk_index}:{chunk_hash_prefix}`

Your labeled dataset should reference `passage_id` values, not raw text. That’s how you can reindex without rewriting your whole eval set.

### Step 5: Store your evaluation run metadata

This is the part people avoid because it feels like process. It’s not. It’s how you stop debates.

LangSmith’s lifecycle framing (offline and online evaluations, attaching evaluators to runs) pushes you toward capturing run context (LangSmith team). Even if you don’t use LangSmith, steal the idea:

- `eval_run_id`
- code commit SHA
- chunker version
- embeddings version
- reranker version
- prompt template version
If your eval can’t answer “what changed?”, it’s not an eval. It’s a horoscope.

(If you’re building broader monitoring around this, my post on [LLM observability metrics](/blog/llm-observability-metrics) and production AI patterns is the companion piece.)

## Citation accuracy and grounding: make it a first-class metric

Most RAG stacks ship “citations” as UI decoration. That’s a mistake.

If you show citations, you can score them. If you can score them, you can catch grounding regressions even when the answer still sounds fluent.

### A concrete citation scoring protocol

Pick one protocol and stick to it. Three options, from easiest to strictest:

1. **Passage-ID match**: The answer cites `passage_id` values. A citation is correct if it points to a relevant passage.
1. **Quote overlap**: The answer includes short quotes; score whether quoted spans exist verbatim in the cited passage.
1. **Span-level attribution**: Every claim sentence is mapped to one passage. Expensive, but the cleanest debugging.
I usually start with passage-ID match because it’s cheap and automatable.

### Metrics I actually track

Per query:

- **Cited-and-retrieved rate**: cited passages are in top `k` retrieval.
- **Cited-but-not-retrieved rate**: the model cites a passage it never saw. That’s a red flag for hallucinated citations or caching bugs.
- **Retrieved-but-not-cited rate**: retrieval found good stuff, generation ignored it.
- **Wrongly-cited rate**: citations exist but don’t support the claim.
If you only track a single “faithfulness” score you’ll miss these failure modes. And those failure modes are exactly what users complain about.

Same mindset applies to agent systems. If you care about correctness under adversarial inputs, you should be running prompt injection regression tests and looking at AI security controls. Citations are not just UX. They’re part of your trust boundary.

## Chunk leakage / contamination tests (the stuff that makes your evals lie)

Chunk leakage is when your evaluation gets artificially inflated because the system “cheats.” Not maliciously. Structurally.

Three common causes:

1. **Overly large chunks** that include the answer text plus surrounding context, making retrieval trivial.
1. **Duplicated boilerplate** (nav bars, repeated footers, “terms of service” blocks) creating near-duplicate chunks across many docs.
1. **Train/test contamination** in offline corpora: the same paragraph lands in both your golden set and your retrieval index due to copy/paste docs, versioned pages, or mirrored content.
### The tests I run

- **Near-duplicate chunk detection**: MinHash / SimHash or embedding cosine similarity to find chunks with similarity > **0.95** across splits.
- **Boilerplate frequency scan**: identify spans that appear in > **5%** of chunks. Remove or downweight.
- **Answer-in-chunk heuristic**: if your labeled answer string appears verbatim inside a chunk, flag it. This catches “the chunk literally contains the final answer” situations.
If you want a security-oriented version of this, I wrote a dedicated [RAG data leakage test suite](/blog/rag-data-leakage-test-suite) and a broader [LLM data leakage playbook](/blog/llm-data-leakage-playbook). Leakage isn’t just an eval integrity issue. It’s a compliance issue.

## Two-stage retrieval (retriever + reranker): evaluate without conflating improvements

In 2026, almost every serious production RAG stack is two-stage:

1. A fast retriever (vector / hybrid) gets top `N` candidates.
1. A reranker reorders them (cross-encoder, LLM reranker, lightweight transformer).
If you only measure “final top-10 quality,” you can’t tell whether:

- the retriever got better
- the reranker got better
- the reranker is quietly masking retriever regressions
### The clean separation

Measure:

- **Retriever recall@N** (large N, like **50** or **100**) against labeled relevant passages.
- **Reranker MRR@k / nDCG@k** on the reranked top-k list.
If retriever recall@100 drops from **0.92 → 0.80**, your reranker might keep MRR@10 stable for a while. Then you hit the query where the relevant passage never makes it into the candidate set. Users feel it immediately.

I’ve shipped systems where Kafka event-streaming the context pipeline mattered more for latency than model-side tricks. In the Walmart chatbot stack, that kind of engineering is why we could keep responses sub-second at scale. The eval analogue is the same. Measure each stage. The bottleneck is rarely the one everyone is staring at.

(If you’re deep into reranking and context limits, this pairs well with [RAG](/glossary/rag) fundamentals and my post on [RAG context window limits](/blog/rag-context-window-limitations).)

## Offline and online evaluations: thresholds, SLOs, and regression alerts

Offline evals are your release gate. Online metrics are your “something broke at 2am” alarm.

### What to evaluate (in production)

At minimum:

- **Recall@10** on your golden set, run on every PR that touches retrieval config.
- **MRR** for reranked results.
- **Citation accuracy** on a smaller curated set.
- **Latency budget** for retrieval and reranking (p50/p95). Even a **100 ms** reranker can destroy your UX.
### Setting “good enough” thresholds

I use three bands:

- **Block**: drop > **2–3 points** on recall@10 or MRR vs last release.
- **Warn**: drop **1 point**, open an investigation.
- **Monitor**: changes under **1 point**, but track the trend.
The numbers depend on domain. The structure doesn’t. “Looks fine” is not a threshold.

### Alerting on retrieval drift

Online, you usually don’t have labels. So you alert on proxies:

- distribution shift in embedding distances
- sudden increase in “no results” rate
- sudden increase in filter-empty results
- spikes in “retrieved-but-not-cited” (if you have citations)
Tracing metadata is non-negotiable here. If you can’t break down metrics by embedding model version or chunker version, you’ll waste days arguing about which change caused the regression.

If you’re building an eval program beyond RAG, see my [agent evaluation roadmap](/blog/agent-evaluation-roadmap-teams) and [AI engineering eval gates](/blog/ai-engineering-evals-gates). Same mindset. Same operational payoff.

Here’s a solid overview video if you want a second perspective on metric choices before you implement:

[Watch: RAG Models Evaluation | Top 12 Metrics for Retrieval Augmented Generation](https://www.youtube.com/watch?v=17YtsJwNecc)

## Response evaluation: how to diagnose “bad retrieval vs bad synthesis vs bad prompt formatting”

Once retrieval is measured, response eval becomes useful instead of vague.

I split response failures into three buckets:

1. **Missing context**: retrieval failed. Recall@k is low. Fix embeddings, chunking, query rewriting, filters, or index params.
1. **Bad synthesis**: retrieval succeeded (high recall@k), but the LLM response is wrong. Fix prompt, model choice, decoding, or add stronger answer constraints.
1. **Bad formatting / tool contract**: the answer is conceptually right but violates schema, tone, or policy. Fix output parsing, schemas, and guardrails.
If you want judge-based metrics, use them like LlamaIndex does. Separately score answer relevancy and context relevancy (LlamaIndex team). The separation is the point.

And please don’t treat “faithfulness” as one monolithic number. Faithfulness is downstream of retrieval quality. If you don’t measure retrieval, you’ll misattribute failures and “fix” the wrong layer.

## Best practices (building datasets) that don’t rot after a month

This is where most teams fail. Not on metric formulas. On discipline.

My practical rules:

- **Start small, grow weekly**: 30 queries to bootstrap, then add **5–20** per week.
- **Stratify by intent**: “definition,” “how-to,” “comparison,” “policy,” “edge cases.” If 80% of your golden set is easy FAQs, your metrics will flatter you.
- **Label passages, not docs**: passage IDs give you stable measurement.
- **Keep a hard ‘canary’ set**: 10–20 queries you never remove. These catch slow drift.
- **Version everything**: docs snapshot, chunker, embeddings, index, reranker.
One more production reality check: cost.

When I built the Azure OpenAI RAG analytics microservice for Firework’s AI short-video platform (2022–2025), the biggest bill surprise wasn’t tokens. It was retries and regeneration. Retrieval regressions increase retries because the model gets less confident, asks follow-ups, or produces low-quality answers that trigger re-asks. Track LLM cost alongside retrieval metrics if you like your margins.

If you want to go deeper into architecture choices before you even get to evals, my LLM knowledge base architecture guide and [fine-tuning vs RAG vs prompt engineering](/blog/fine-tuning-vs-rag-prompt-engineering) posts are the “pick the right approach” side of this.

By 2027, “RAG evals” won’t look like a single score. They’ll look like a contract suite. Retrieval quality, citation grounding, leakage checks, and stage-by-stage metrics, treated like CI tests.

If your team is still grading answers and calling it evaluation, you’re going to keep learning about regressions from customers first. Stop doing that. Build the harness.

Photo by Justin Morgan on Unsplash.

## FAQ

### What metrics should I use to evaluate a RAG system?

Use two sets of metrics: retrieval metrics and response metrics. For retrieval, start with recall@k, precision@k, hit rate@k, and MRR, then add nDCG if you need graded relevance. For responses, add a grounding or faithfulness metric plus a simple correctness check, but only after you can measure retrieval quality independently.

### How do you create a gold dataset for RAG evaluation?

Freeze a document snapshot, chunk it with a versioned chunker configuration, and assign stable passage IDs. Then create a set of real queries and label the relevant passage IDs for each query, allowing multiple relevant passages when needed. Keep the dataset immutable for regression testing and grow it slowly over time.

### What is the difference between retrieval evaluation and answer evaluation?

Retrieval evaluation measures whether the system fetched the right passages, typically using recall@k, MRR, or nDCG on a labeled dataset. Answer evaluation measures whether the final generated response is correct, helpful, and properly grounded. You need both, but retrieval evaluation is what tells you why a RAG system regressed after index, embedding, or chunking changes.