Handling no results, conflicting sources & stale data

The mental model: RAG failure modes as a pipeline control flow problem

Think of retrieval as a gated pipeline with three checkpoints before generation. Checkpoint one: did we retrieve anything with a score above the minimum acceptable relevance? Checkpoint two: are the retrieved chunks mutually consistent, or do they contradict each other on key facts? Checkpoint three: is the retrieved content still temporally valid? Each checkpoint has a pass, a warn, and a fail path. Most teams wire up only the happy path (pass) and send everything else straight to the LLM. The LLM then hallucinates a confident-sounding answer from bad input, and the user blames the AI when the real failure was in the pipeline logic upstream.

Similarity scores from vector search (cosine, dot product, or L2 depending on your database) are noisy signals. A chunk scoring 0.61 in a Qdrant search might be genuinely relevant for a narrow technical query, or it might be a tangentially related paragraph that shares vocabulary but not meaning. You cannot pick a universal threshold. Instead, calibrate per collection or per query domain by sampling 50-100 representative queries, manually labeling the top-3 results, and plotting score vs. relevance. You'll usually find a natural inflection point. For OpenAI text-embedding-3-small, scores above roughly 0.75 are usually safe; below 0.55 is noise. But those are illustrative starting points, not production config.

Real-world scenario: a B2B SaaS internal knowledge base

Imagine a company uses RAG over its internal Confluence and Notion exports to answer employee questions. Three problems surface in the first month. First, questions about new product features return zero results because the docs haven't been ingested yet, and the LLM invents answers. Second, the pricing page was updated, but the old version is still in the index alongside the new one. Employees get a mix of old and new pricing in the same response. Third, a regulatory compliance doc was updated 14 months ago but is indexed without a timestamp, so the pipeline treats it as authoritative.

A senior engineer approaches this methodically. For no-results: they set a threshold of 0.70, and when all retrieved chunks fall below it, they trigger query rewriting via a small, cheap LLM call (GPT-4o-mini or Claude Haiku) that expands acronyms, adds synonyms, and strips filler words. If the rewritten query also fails, the pipeline returns a structured fallback message that says what it searched for and suggests the user contact the relevant team. For conflicts: they store source_url, doc_version, and ingested_at in the metadata at ingestion time. Before sending chunks to generation, a comparison step checks if two or more chunks cover the same entity (same URL prefix or same doc ID) but disagree on numeric values or boolean facts. If they do, they pick the chunk with the most recent ingested_at and annotate the response with a warning that a conflict was detected. For staleness: they set a TTL of 30 days per document type (7 days for pricing pages, 90 days for architecture docs), and a background job checks ingested_at against TTL nightly and flags expired documents for re-ingestion.

Tradeoffs vs alternative approaches

Query rewriting adds a round-trip LLM call (50-150ms and a few hundred tokens). The alternative is retrieval with HyDE (Hypothetical Document Embeddings), where you generate a fake answer and embed it as the query. HyDE often retrieves better on vague queries but is slower and burns more tokens per request. For no-results fallback, some teams fall back to a pure LLM response without context. That works if your LLM has strong world knowledge about the domain, but it breaks the RAG contract: users expect answers grounded in your documents. A middle ground is to fall back to a broader BM25 keyword search before giving up on retrieval entirely, since BM25 handles exact-match queries that dense retrieval misses.

For conflict detection, you can either do it in the pipeline (pre-generation) or in the LLM prompt (post-generation). Pre-generation conflict detection is more reliable because you can act on it deterministically. Post-generation detection (instructing the LLM to flag contradictions) is probabilistic and depends on the model noticing the conflict in context, which it doesn't always do.

What changes at scale

At 10 users, you can re-index everything nightly and do conflict detection synchronously in the request path. At 10k users, nightly full re-index is too slow for dynamic content. You need an event-driven ingestion pipeline: a webhook or change-data-capture (CDC) stream from your source system triggers incremental upserts to the vector store. Conflict detection starts to be too expensive per request if you're running an LLM call for it. Move to a metadata-only heuristic: sort chunks by ingested_at descending, take only the most recent per source document. At 10M users, your vector database query latency becomes the bottleneck. You'll want read replicas, sharded collections by domain, and a cache layer (Redis with TTL-matched to your staleness budget) for queries that repeat within a short window. The no-results fallback path must be async and cheap because it's now handling thousands of requests per minute.