AI & ML

Retrieval-Augmented Generation (RAG)

A technique that enhances LLM responses by retrieving relevant documents from an external knowledge base before generating an answer.

Retrieval-Augmented Generation (RAG) combines the power of large language models with external knowledge retrieval to produce more accurate, up-to-date, and grounded responses.

In a typical RAG pipeline, a user query is first converted into a vector embedding, which is used to search a vector database for semantically similar documents. The retrieved documents are then injected into the LLM's prompt as context, allowing the model to generate responses grounded in factual, domain-specific information.

RAG solves several key LLM limitations: hallucination (making up facts), knowledge cutoff dates, and lack of domain-specific expertise. It's widely used in enterprise chatbots, customer support systems, and internal knowledge assistants.

Key components of a RAG system include a document ingestion pipeline, a chunking strategy, an embedding model, a vector store, and a retrieval strategy (e.g., semantic search, hybrid search, or re-ranking).

How it works

The term comes from Lewis et al. (2020), who described models that combine parametric memory (knowledge stored in a model's weights) with non-parametric memory (in their case, a dense vector index of Wikipedia accessed by a neural retriever). They noted that updating a model's world knowledge and showing where an answer came from are hard for parametric-only models.

The usual pipeline starts offline. Documents are split into chunks, which Anthropic's description of standard RAG puts at usually no more than a few hundred tokens each; every chunk is embedded, and the vectors are stored in a vector database that can search by semantic similarity.

At question time the query is embedded, the closest chunks are retrieved, and they are placed into the prompt with instructions to answer from them and cite them. The language model then writes an answer grounded in that retrieved text rather than only in what it memorised during training.

Answer quality depends heavily on retrieval. In Anthropic's published contextual-retrieval experiments, adding chunk-specific context to both embeddings and BM25 cut the top-20 retrieval failure rate by 49% (5.7% to 2.9%), and adding a reranking step cut it by 67% (to 1.9%).

Example

The core of a RAG request in Python-style pseudocode. embed, vector_db and llm stand in for whichever embedding model, store and LLM client are used.

question = "What is the refund window for annual plans?"
chunks = vector_db.search(embed(question), top_k=4)

prompt = (
    "Answer using only the sources below and cite the source id.\n"
    "If the answer is not in the sources, say so.\n\n"
    + "\n\n".join(f"[{c.id}] {c.text}" for c in chunks)
    + f"\n\nQuestion: {question}"
)
answer = llm.generate(prompt)

If the refund policy changes, only the source document needs re-indexing; the model is untouched. If retrieval returns the wrong four chunks, no prompt wording will rescue the answer, which is why retrieval quality is usually the first thing to measure.

Common confusion

What is the difference between RAG and fine-tuning?

RAG changes what the model sees at question time by adding retrieved documents to the prompt; the weights stay the same, and updating knowledge means re-indexing documents. Fine-tuning changes the weights by training on examples, which suits teaching a style, format or skill, but makes facts harder to update and to trace to a source. The two are often combined.

Is RAG still needed with long context windows?

Long context windows let more text fit into one prompt, but every token sent is paid for and processed on every request. Retrieval selects the few passages that matter, keeping prompts smaller and cheaper, and works for corpora far larger than any context window. For a small, fixed set of documents, putting all of them in the prompt can be the simpler choice.

RAG Cost Calculator Estimate embedding, storage and generation costs for a RAG pipeline.
Embedding Space Explorer See how retrieval finds nearby meanings.
AI Token Counter Check how many tokens your retrieved chunks add to each prompt.
RAG Architecture Playbook Chunking, retrieval and evaluation choices in more depth.
Vector Database Where the chunk embeddings live.