Semantic Search

At indexing time every document, or every chunk of a long document, is passed through an embedding model and the resulting vector is stored. At query time the query is embedded with the same model and the system returns the documents whose vectors are closest, most often by cosine similarity.

Comparing the query against every stored vector is fine for modest corpora; the sentence-transformers semantic_search utility is documented for corpora up to about a million entries. Larger systems use an approximate nearest-neighbour index inside a vector database, trading a little recall for much faster lookups.

The right model depends on the shape of the task. The sentence-transformers documentation distinguishes symmetric search, where query and documents are similar in length (finding duplicate questions), from asymmetric search, where a short query must match a longer passage, and stresses choosing a model trained for the right one.

Semantic search is often combined with lexical scoring. BM25, Elasticsearch's default, scores documents by term frequency, inverse document frequency and document-length normalisation (default k1 = 1.2, b = 0.75). Lexical matching catches exact identifiers such as error codes or product names that embeddings can blur, and the merged results can then be reordered by a reranking model.

Embedding a three-sentence corpus and a query with sentence-transformers, following the library's own semantic search example.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
corpus = [
    "Rolling back a Kubernetes deployment",
    "Baking sourdough bread at home",
    "Restoring the previous version of a release",
]
corpus_emb = model.encode_document(corpus, convert_to_tensor=True)
query_emb = model.encode_query("how do I undo a bad deploy", convert_to_tensor=True)

scores = model.similarity(query_emb, corpus_emb)[0]
for text, score in sorted(zip(corpus, scores.tolist()), key=lambda p: -p[1]):
    print(f"{score:.3f}  {text}")

The query shares no words with "Restoring the previous version of a release", yet that sentence and the Kubernetes rollback are expected to score well above the bread recipe, because the model places them near each other in meaning. A pure keyword search for "undo" would match none of the three. Run it to see the actual scores for this model.

What is the difference between semantic search and keyword search?

Keyword search (for example BM25) ranks documents by how often the query's exact terms appear, weighted by how rare those terms are. Semantic search ranks by closeness of meaning using embeddings, so it finds paraphrases and synonyms but can miss exact strings like version numbers. Hybrid search runs both and merges the results.

Is semantic search the same as RAG?

No. Semantic search returns documents. Retrieval-augmented generation takes the retrieved documents and puts them into an LLM prompt so the model can write an answer grounded in them. Semantic search is frequently the retrieval step inside a RAG system, but it is useful on its own as a search box.