BM25 is a bag-of-words ranking function built on TF-IDF logic. It scores a document higher when query terms appear frequently in that document but rarely across the whole corpus (that's the IDF component), and it applies a saturation curve so a term appearing 20 times doesn't score 20x better than one appearing twice. What BM25 cannot do is recognize that "automobile" and "car" are related, or that a query for "how do I scale horizontally" should surface a document about "sharding and replication." Vector search solves exactly that problem by projecting both the query and documents into a shared semantic space where proximity encodes meaning. The fundamental insight behind hybrid search is that these two failure modes are almost perfectly complementary, so combining them reduces both.
Reciprocal Rank Fusion is the standard way to merge result lists. For each document, you look at its rank in the BM25 list and its rank in the vector list, compute 1/(k + rank) for each (where k is a smoothing constant, conventionally 60), and sum those values. The constant k prevents the top-ranked document from getting an outsized score and dampens the effect of rank differences near the top. The critical property of RRF is that it operates on ranks, not raw scores. BM25 returns floating-point values in some arbitrary range; cosine similarity returns values between -1 and 1. Trying to linearly combine those directly requires careful normalization that breaks every time your corpus changes. RRF sidesteps this entirely.
Consider a customer support system for a SaaS product. A user asks: "Why does my Webhook event PAYMENT_FAILED_3047 not fire on retry?" This query has a semantic component (webhook retry behavior) and a very specific keyword component (PAYMENT_FAILED_3047). Vector search will return documents about webhook retry logic, idempotency, and event delivery, which are semantically correct but may not contain that exact error code. BM25 will surface any document containing PAYMENT_FAILED_3047, but may rank a changelog entry about that code above the actual troubleshooting guide. Hybrid search returns the troubleshooting guide at the top because it scores well on both dimensions. This is the canonical use case where hybrid search pays off immediately.
Versus pure vector search, hybrid adds BM25 indexing overhead at both ingestion and query time. BM25 is lightweight, but maintaining two indexes introduces operational complexity. Versus a learning-to-rank (LTR) model or a re-ranker, hybrid search is simpler to set up and doesn't require labeled training data. Re-ranking (a separate sibling subtopic) is often applied on top of hybrid retrieval, not instead of it: you use hybrid to get a broad candidate set of 50-100 documents, then pass those to a cross-encoder re-ranker to produce the final top-5 or top-10. These layers are additive, not exclusive.
At 10 users, you can run BM25 fully in-memory with rank-bm25 and keep vector search in a small Qdrant or pgvector instance. At 10,000 users, latency becomes the concern: BM25 over a large corpus is O(n) per query unless you have an inverted index. Elasticsearch and Weaviate both use inverted indexes internally, which reduces BM25 query time to O(k) where k is the number of matching documents. At 10 million users, you need both to be distributed and you need to decide whether to run hybrid at the retrieval layer or at an intermediate fusion service. Weaviate's native hybrid search, Pinecone's sparse-dense index, and Elasticsearch's built-in RRF (added in 8.8) all handle this without you building a custom fusion service. One often-overlooked cost implication: sparse vectors (BM25-style) stored in Pinecone's hybrid mode are billed differently from dense vectors, so benchmark your actual corpus before committing to a managed service.
Key Takeaways
- Run vector search and BM25 in parallel, then fuse results using Reciprocal Rank Fusion.
- Use hybrid search when queries mix semantic intent with exact identifiers, acronyms, or rare terms.
- RRF uses rank position, not raw scores, so it handles mismatched scoring scales automatically.
- Tune the alpha weight between vector and keyword scores per query type, not globally.
Pro tips
- The k=60 constant in RRF was chosen empirically in the original paper for web search. For short corpora (under 10k documents), try k=10 to 20. Higher k flattens the score distribution and benefits documents that rank mid-tier in both lists.
- BM25 is sensitive to tokenization. If your corpus contains camelCase identifiers like getUserById or hyphenated model names like gpt-4o, split them before indexing. A BM25 index built on unsplit tokens will miss queries that split them, and vice versa.
- When using Weaviate's alpha parameter for hybrid search, alpha=0.75 weights vector search heavily. Don't set this globally. For queries that look like natural language, use higher alpha; for queries containing numbers, version strings, or identifiers, lower it. You can classify query type with a small heuristic before calling the search API.
- If your vector database charges per query (Pinecone serverless model, for example), BM25 pre-filtering is a cheap way to reduce the candidate set before vector search. Run BM25 first, take the top 200 doc IDs, then run vector search with a filter scoped to those IDs. This can cut vector query costs significantly on large corpora.
Common pitfalls
- Mistake: Linearly combining raw BM25 and cosine scores with a fixed weight. Fix: Use RRF or normalize both score distributions to [0,1] per query before weighting; raw scales are incompatible and change as the corpus grows.
- Mistake: Rebuilding the BM25 index from scratch on every query. Fix: Build the index once at startup or cache it; BM25Okapi construction is O(n * avg_doc_length) and becomes a bottleneck above ~50k documents.
- Mistake: Using the same chunking for both BM25 and vector indexes. Fix: BM25 benefits from slightly larger chunks with more term overlap; vector search benefits from smaller, focused chunks. Consider dual chunking strategies if retrieval quality matters.
- Mistake: Ignoring that BM25 scores zero for documents with no query term overlap. Fix: This is expected but dangerous when all BM25 scores are zero (pure semantic query). Detect this case and fall back to vector-only rather than letting zero-score BM25 documents drag down fused rankings.
When to use hybrid search vs pure vector vs pure BM25
| Option | Use when | Avoid when |
|---|---|---|
| Pure vector search | Queries are natural language, corpus uses consistent vocabulary, users paraphrase frequently. | Corpus contains exact identifiers, product codes, version numbers, or domain jargon not in embedding training data. |
| Pure BM25 | Queries are keyword-driven (log search, SKU lookup), latency budget is very tight, corpus changes in real time. | Users phrase queries differently from how documents are written, or synonyms and paraphrasing are common. |
| Hybrid search with RRF | Queries mix natural language intent with specific identifiers; production RAG systems where retrieval quality matters most. | Operational complexity of maintaining two indexes is not justified, e.g., prototype or corpus under 1k documents. |
| Hybrid + re-ranker on top | You need high precision in the final top-3 to top-5 results and can afford 100-300ms extra latency for a cross-encoder pass. | Latency budget is under 200ms end-to-end or you lack labeled data to evaluate re-ranker quality on your domain. |
Code Example
# requires: rank-bm25==0.2.2, sentence-transformers==2.7.0, numpy==1.26
import numpy as np
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity
corpus = [
"K-Means is a clustering algorithm that partitions data into k groups.",
"Product SKU-9921 has a retail price of 49.99 USD.",
"Hierarchical clustering builds nested clusters using a dendrogram.",
"SKU-1042 is currently out of stock in all warehouses.",
]
query = "clustering methods for grouping data"
# BM25 retrieval
tokenized_corpus = [doc.split() for doc in corpus]
bm25 = BM25Okapi(tokenized_corpus)
bm25_scores = bm25.get_scores(query.split())
# Vector retrieval
model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = model.encode(corpus)
query_embedding = model.encode([query])
vector_scores = cosine_similarity(query_embedding, embeddings)[0]
# Reciprocal Rank Fusion (RRF) with k=60
def rrf_score(scores, k=60):
ranks = np.argsort(scores)[::-1]
rrf = np.zeros(len(scores))
for rank, idx in enumerate(ranks):
rrf[idx] = 1.0 / (k + rank + 1)
return rrf
fused = rrf_score(bm25_scores) + rrf_score(vector_scores)
for idx in np.argsort(fused)[::-1]:
print(f"{fused[idx]:.4f} | {corpus[idx]}")How this code works
This code demonstrates "hybrid search," merging traditional keyword matching with modern semantic understanding to find relevant documents within a corpus. It starts by defining a corpus (a list of documents) and a query. For keyword matching, the BM25Okapi algorithm processes the corpus, splitting each document into individual words using doc.split(). This creates a tokenized_corpus which bm25.get_scores then uses to rank documents based on how well their keywords match the query. This approach is excellent for finding exact keyword hits.
Next, the code employs SentenceTransformer("all-MiniLM-L6-v2") to perform semantic search. This model converts both the corpus documents and the query into numerical embeddings, which are vector representations capturing their meaning. cosine_similarity then measures the "distance" between the query's embedding and each document's embedding, yielding vector_scores for semantic relevance. To combine these two disparate ranking methods, rrf_score implements Reciprocal Rank Fusion. This function's k=60 parameter is a subtle but important detail: it's an offset that moderates the impact of lower-ranked items, preventing a single high rank from dominating, especially when one method might return many zero scores for irrelevant documents. Finally, the total fused scores are used to sort and print the most relevant documents.
Production-grade example
Adds retries, timeouts, structured logging, graceful degradation to BM25-only when vector search fails.
# requires: qdrant-client==1.9.1, rank-bm25==0.2.2, openai==1.30.0, tenacity==8.3.0
import os
import time
import logging
from typing import Optional
from rank_bm25 import BM25Okapi
from qdrant_client import QdrantClient
from qdrant_client.http.models import ScoredPoint
from openai import OpenAI, APIError, RateLimitError
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
logger = logging.getLogger(__name__)
openai_client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
qdrant = QdrantClient(url=os.environ["QDRANT_URL"], api_key=os.environ.get("QDRANT_API_KEY"), timeout=5.0)
@retry(
retry=retry_if_exception_type((RateLimitError, APIError)),
wait=wait_exponential(multiplier=1, min=2, max=30),
stop=stop_after_attempt(4),
)
def embed_query(text: str) -> list[float]:
start = time.perf_counter()
response = openai_client.embeddings.create(
model="text-embedding-3-small",
input=text,
timeout=8.0,
)
latency_ms = (time.perf_counter() - start) * 1000
tokens_used = response.usage.total_tokens
logger.info("embed_query tokens=%d latency_ms=%.1f", tokens_used, latency_ms)
return response.data[0].embedding
def bm25_search(corpus: list[str], query: str, top_k: int = 20) -> list[tuple[int, float]]:
tokenized = [doc.lower().split() for doc in corpus]
bm25 = BM25Okapi(tokenized)
scores = bm25.get_scores(query.lower().split())
ranked = sorted(enumerate(scores), key=lambda x: x[1], reverse=True)[:top_k]
return ranked
def vector_search(collection: str, query_vector: list[float], top_k: int = 20) -> list[ScoredPoint]:
try:
return qdrant.search(collection_name=collection, query_vector=query_vector, limit=top_k, with_payload=True)
except Exception as exc:
logger.error("vector_search failed collection=%s error=%s", collection, exc)
return [] # graceful degradation: fall back to BM25-only
def reciprocal_rank_fusion(
bm25_results: list[tuple[int, float]],
vector_results: list[ScoredPoint],
corpus_ids: list[str],
k: int = 60,
) -> list[tuple[str, float]]:
scores: dict[str, float] = {}
for rank, (doc_idx, _) in enumerate(bm25_results):
doc_id = corpus_ids[doc_idx]
scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
for rank, hit in enumerate(vector_results):
doc_id = hit.id if isinstance(hit.id, str) else str(hit.id)
scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
return sorted(scores.items(), key=lambda x: x[1], reverse=True)
def hybrid_search(
query: str,
corpus: list[str],
corpus_ids: list[str],
collection: str,
top_k: int = 5,
fallback_to_bm25: bool = True,
) -> list[dict]:
query_vector: Optional[list[float]] = None
try:
query_vector = embed_query(query)
except Exception as exc:
logger.warning("embed_query exhausted retries, using BM25-only fallback error=%s", exc)
bm25_results = bm25_search(corpus, query, top_k=20)
vector_results = vector_search(collection, query_vector, top_k=20) if query_vector else []
if not vector_results and not fallback_to_bm25:
raise RuntimeError("Both retrieval methods failed")
fused = reciprocal_rank_fusion(bm25_results, vector_results, corpus_ids)
logger.info("hybrid_search query=%r bm25_hits=%d vector_hits=%d fused_top5=%s",
query[:60], len(bm25_results), len(vector_results),
[doc_id for doc_id, _ in fused[:top_k]])
return [{"id": doc_id, "rrf_score": score} for doc_id, score in fused[:top_k]]How this code works
This code performs a hybrid search, a crucial technique in advanced RAG (Retrieval Augmented Generation) to find the most relevant documents for a query. It combines two powerful methods: keyword matching and semantic similarity, ensuring both broad recall and precise relevance.
The process starts with embed_query, which converts the user's text query into a numerical vector using OpenAI, capable of understanding the query's meaning. This function includes a @retry decorator for robustness against temporary API errors. Simultaneously, bm25_search performs a traditional keyword-based ranking of documents in the corpus. If the vector embedding is successful, vector_search then queries a Qdrant database to find documents whose vectors are semantically similar to the query. A subtle but important detail is how hybrid_search orchestrates this: if embed_query or vector_search fails (e.g., API issues), it gracefully falls back to using bm25_search only, preventing a complete system failure. Finally, reciprocal_rank_fusion intelligently merges the results from both the keyword and vector searches, giving higher scores to documents that ranked well in both methods to produce the ultimate top_k results.
Practice & master
Try the exercise, check your understanding, then mark this lesson mastered to track your path to pro.
Exercise
Build a hybrid search function that queries a small in-memory corpus using both BM25 and vector similarity, then fuses the results with RRF. Test it against two queries: one semantic ('methods for grouping similar data') and one keyword-specific ('error code E_CONN_TIMEOUT'). Print the top-3 results for each query and confirm the right documents surface.
# requires: rank-bm25==0.2.2, sentence-transformers==2.7.0
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
corpus = [
"K-Means and DBSCAN are popular clustering algorithms for grouping similar data points.",
"Error code E_CONN_TIMEOUT occurs when the TCP connection exceeds the configured timeout.",
"Hierarchical clustering produces dendrograms showing nested group structure.",
"E_CONN_TIMEOUT can be resolved by increasing socket_timeout in the client config.",
"Principal component analysis reduces dimensionality before clustering.",
]
model = SentenceTransformer("all-MiniLM-L6-v2")
corpus_embeddings = model.encode(corpus)
def bm25_ranks(query: str, top_k: int = len(corpus)) -> list[int]:
# TODO: tokenize corpus and query, build BM25Okapi, return sorted doc indices
pass
def vector_ranks(query: str, top_k: int = len(corpus)) -> list[int]:
# TODO: embed query, compute cosine similarity, return sorted doc indices
pass
def rrf_fuse(bm25_order: list[int], vector_order: list[int], k: int = 60) -> list[tuple[int, float]]:
# TODO: compute RRF scores for each doc index, return sorted (idx, score) pairs
pass
for query in ["methods for grouping similar data", "error code E_CONN_TIMEOUT"]:
bm25_order = bm25_ranks(query)
vector_order = vector_ranks(query)
fused = rrf_fuse(bm25_order, vector_order)
print(f"\nQuery: {query}")
for idx, score in fused[:3]:
print(f" {score:.4f} | {corpus[idx]}")Quick check
Why does Reciprocal Rank Fusion use rank position instead of combining raw BM25 and cosine scores directly?
A user queries your RAG system for 'SKU-7741 return policy.' Pure vector search returns shipping and returns policy docs but not the SKU-7741 specific document. What does this tell you?
You set k=60 in RRF and find that documents ranking 1st and 2nd in both lists get almost identical fused scores. What change would increase the spread between top-ranked and lower-ranked documents?