How similarity math actually works
An embedding model converts text into a vector -- an ordered list of floats, typically 768 to 3072 dimensions. The core intuition: semantically similar text lands close together in that high-dimensional space. "Closeness" is defined by a distance function. Cosine similarity measures the angle between two vectors. If both vectors point in the same direction, the angle is 0 and cosine is 1. If they point opposite directions, cosine is -1. Crucially, cosine ignores magnitude -- a 500-word essay and a 5-word sentence on the same topic can score close to 1.0. The dot product (also called inner product) is sum(a_i * b_i) across all dimensions. It mixes both angle and magnitude. For a unit-length vector where ||v|| = 1, cosine similarity and dot product are mathematically identical: cos(a,b) = dot(a,b) / (||a|| * ||b||), and when both norms are 1 that denominator vanishes. Most modern embedding APIs (OpenAI text-embedding-3-*, Cohere embed-v3) return L2-normalized vectors by default. That means dot product is faster (one fewer division) and gives the same ranking. Euclidean distance (L2) is a third option but less common for text; it's sensitive to magnitude and can rank long documents differently than short ones even on the same topic.
A real RAG scenario
You're building a support bot over 50,000 documentation pages. A user asks: "How do I rotate API keys?" You embed the question with the same model used to index the docs, then retrieve the top-5 most similar chunks. Those chunks become the context passed to GPT-4o. The entire retrieval step needs to complete in under 100ms for the full pipeline to feel snappy. With 50k vectors at 1536 dimensions, brute-force comparison costs roughly 50k dot products -- fast on a GPU, borderline on a CPU in a serverless function that has to cold-start. A senior engineer would reach for an ANN index here, cache the index in memory between requests, and set recall@10 target (not recall@5) to give the LLM options.
Exact vs. Approximate Nearest Neighbors
Exact KNN scans every vector and guarantees the true nearest neighbors. It scales as O(n * d) per query where n is corpus size and d is dimension. At 100k vectors it's fine. At 10M it's unusable at interactive latency. ANN algorithms build an index structure offline so queries scan a fraction of the dataset. HNSW (Hierarchical Navigable Small World) is the dominant choice today: it builds a multi-layer graph where each layer is a sparser version of the one below. Query traversal starts at the top (few nodes, coarse navigation) and drills down. It achieves ~95-99% recall at 10-50x speedup over brute-force, depending on dataset. IVF (Inverted File Index, the approach FAISS's IVFFlat uses) clusters vectors with k-means first, then at query time only searches the nearest clusters. Faster to build than HNSW but typically lower recall for the same speed budget. ScaNN (Google) uses learned quantization and is strong on recall-per-CPU-cycle benchmarks. In practice: use HNSW as your default, reach for IVF+PQ when memory is constrained.
Tradeoffs between approaches
Exact search has one advantage: it's simple and has no recall-vs-speed knob to tune. Use it when n < 50k or in offline batch jobs where latency doesn't matter. HNSW has two key parameters: M (graph connectivity, higher = better recall, more memory) and ef_construction (build-time quality, higher = slower build, better index). At query time ef_search controls the recall-speed tradeoff dynamically. A common mistake is leaving these at library defaults -- the defaults are conservative. Benchmark your dataset and push ef_search up until recall@10 hits 0.95+, then check whether p99 latency is still acceptable. Product Quantization (PQ) compresses vectors from 32-bit floats to 4-8 bits per dimension, reducing memory 4-8x at the cost of ~2-5% recall drop. At 10M+ vectors this is often the only way to keep the index in RAM.
What changes at scale
At 10 users: exact numpy search, no database needed, entire corpus in memory. At 10k users with 1M vectors: you need a persistent vector store with a proper ANN index. Qdrant, Weaviate, and Pinecone all use HNSW under the hood. Latency is dominated by network round-trip to the vector DB (typically 5-20ms) plus index traversal. At 10M users with 100M+ vectors: sharding matters. Most managed vector databases shard automatically, but you need to think about whether your data has a natural partition key (by tenant, by language, by date) that lets you limit search scope per query. Multi-tenancy with namespace isolation is often cheaper than one giant shared index. Also at this scale: batch embedding writes, async indexing pipelines, and pre-filtering (metadata filters applied before or during ANN search) all become critical for cost control.
Key Takeaways
- Use cosine similarity for normalized embeddings; dot product is equivalent and faster when vectors are already unit-length.
- Brute-force exact search is fine under ~100k vectors; above that, switch to an ANN index.
- ANN algorithms trade a small recall loss for orders-of-magnitude speedup -- tune ef/nprobe to control that tradeoff.
- Always benchmark recall@k on your own data before choosing an index type or similarity metric.
Pro tips
- When you normalize vectors to unit length before storing them, dot product and cosine similarity give identical rankings -- but dot product skips the denominator division, making batch queries measurably faster at 10M+ scale. Most OpenAI and Cohere models return normalized vectors; verify with
np.linalg.norm(vec)close to 1.0 before assuming. - hnswlib's
ef_searchparameter can be changed at runtime without rebuilding the index. In practice, set a low value for bulk batch queries (speed priority) and a high value for interactive user queries (recall priority), and expose it as a per-request parameter. - Recall@k is not the same as precision. A retrieval step that returns the correct document somewhere in the top-20 can still power a good RAG answer if the LLM is reading all 20. Set
ef_searchto hit recall@20 > 0.97 rather than recall@5 > 0.97 -- it's usually cheaper in terms of index parameters. - Pre-filtering (applying metadata filters before ANN search) can silently destroy recall if the filtered subset is small -- the ANN graph was built on the full corpus and navigates poorly over a sparse subgraph. Qdrant and Weaviate handle this with post-filtering or payload indexes; understand which strategy your vector DB uses before relying on filtered search.
Common pitfalls
- Mistake: Using cosine similarity with unnormalized vectors from a model that doesn't normalize by default. Fix: Always normalize (
v / ||v||) before indexing or querying, or verify the model spec explicitly states L2-normalized output. - Mistake: Leaving HNSW
ef_constructionandMat library defaults, then wondering why recall is 85%. Fix: Benchmark recall@10 on a held-out sample; increaseMto 32-64 andef_constructionto 200+ for better quality. - Mistake: Running exact brute-force KNN in a synchronous API handler at 500k+ vectors. Fix: Build an ANN index once at startup, keep it in process memory, and serve queries from it -- this cuts p99 latency from seconds to single-digit milliseconds.
- Mistake: Mixing embedding models between indexing and query time (e.g., indexing with text-embedding-ada-002, querying with text-embedding-3-small). Fix: Pin the model ID in a config constant and enforce it at both write and read paths with an assertion.
When to use cosine similarity vs dot product vs L2 distance
| Option | Use when | Avoid when |
|---|---|---|
| Cosine similarity | Vectors may not be unit-length and you care only about semantic direction, not magnitude. | Vectors are already normalized -- you're paying for a division you don't need. |
| Dot product (inner product) | Vectors are L2-normalized (most modern embedding APIs). Mathematically equivalent to cosine but faster. | Vectors are not normalized and magnitude encodes no useful information -- results will be skewed by vector length. |
| Euclidean (L2) distance | You're working with image embeddings or other modalities where absolute position in vector space matters, not just direction. | Text embeddings from transformer models -- L2 behaves poorly compared to cosine for high-dimensional text vectors. |
| Exact KNN (brute-force) | Corpus is under ~100k vectors or you're running an offline batch job where latency is irrelevant. | Interactive queries over 100k+ vectors -- per-query latency will be unacceptable. |
| ANN (HNSW, IVF) | Corpus exceeds 100k vectors and you need sub-100ms query latency in production. | You need 100% guaranteed recall (e.g., legal or compliance search where missing a document has consequences). |
Code Example
# numpy 1.26, no vector DB needed -- pure similarity math
import numpy as np
def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
def dot_product(a: np.ndarray, b: np.ndarray) -> float:
return float(np.dot(a, b))
# Simulate 3 document embeddings (dim=4 for readability)
docs = np.array([
[0.9, 0.1, 0.2, 0.0],
[0.1, 0.8, 0.1, 0.5],
[0.85, 0.15, 0.25, 0.05],
], dtype=np.float32)
query = np.array([0.88, 0.12, 0.22, 0.02], dtype=np.float32)
for i, doc in enumerate(docs):
cos = cosine_similarity(query, doc)
dot = dot_product(query, doc)
print(f"Doc {i}: cosine={cos:.4f} dot={dot:.4f}")
How this code works
This code demonstrates how to calculate the similarity between different data points, specifically "document embeddings" and a "query," using two common methods: cosine similarity and dot product. It uses the numpy library for efficient array operations, mimicking how vector search works without needing a full vector database.
The cosine_similarity function calculates how similar two vectors are by dividing their np.dot product by the product of their lengths (np.linalg.norm). This normalizes the result, making it insensitive to vector magnitude. The dot_product function simply computes np.dot(a, b). Notice that dot_product explicitly converts its result to a standard Python float, even though np.dot might return a numpy.float32 type. This ensures consistent type handling across the code, which can be a subtle point when mixing numpy types with Python's native ones. The code then sets up example docs (documents) and a query as np.arrays, simulating embeddings with four dimensions for clarity. It iterates through each doc using a for i, doc in enumerate(docs): loop, calculates both cos and dot similarities against the query, and finally prints the results for comparison, rounded to four decimal places for readability.
Production-grade example
Adds dimension validation, zero-norm guards, slow-query warnings, structured logging, and retry logic.
# hnswlib 0.7.0, structlog, tenacity
import os
import time
import logging
from typing import Optional
import numpy as np
import hnswlib
import structlog
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
log = structlog.get_logger()
DIM = int(os.environ.get("EMBED_DIM", "1536"))
MAX_ELEMENTS = int(os.environ.get("HNSW_MAX_ELEMENTS", "500000"))
EF_CONSTRUCTION = int(os.environ.get("HNSW_EF_CONSTRUCTION", "200"))
M = int(os.environ.get("HNSW_M", "32"))
EF_SEARCH = int(os.environ.get("HNSW_EF_SEARCH", "100"))
INDEX_PATH = os.environ.get("HNSW_INDEX_PATH", "/tmp/hnsw.bin")
class VectorIndex:
def __init__(self):
self.index = hnswlib.Index(space="cosine", dim=DIM)
self.index.init_index(max_elements=MAX_ELEMENTS, ef_construction=EF_CONSTRUCTION, M=M)
self.index.set_ef(EF_SEARCH)
log.info("hnsw_index_initialized", dim=DIM, M=M, ef_construction=EF_CONSTRUCTION, ef_search=EF_SEARCH)
def add_batch(self, vectors: np.ndarray, ids: list[int]) -> None:
if vectors.shape[1] != DIM:
raise ValueError(f"Expected dim={DIM}, got {vectors.shape[1]}")
if vectors.dtype != np.float32:
vectors = vectors.astype(np.float32)
norms = np.linalg.norm(vectors, axis=1, keepdims=True)
if np.any(norms < 1e-8):
raise ValueError("Zero-norm vector detected -- embedding likely failed")
start = time.perf_counter()
self.index.add_items(vectors, ids)
elapsed_ms = (time.perf_counter() - start) * 1000
log.info("vectors_indexed", count=len(ids), elapsed_ms=round(elapsed_ms, 2))
@retry(
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=0.1, min=0.1, max=2),
retry=retry_if_exception_type(RuntimeError),
)
def query(
self, vector: np.ndarray, k: int = 10, timeout_ms: float = 50.0
) -> tuple[list[int], list[float]]:
if vector.dtype != np.float32:
vector = vector.astype(np.float32)
norm = np.linalg.norm(vector)
if norm < 1e-8:
raise ValueError("Query vector has zero norm")
vector = vector / norm # ensure unit-length for cosine correctness
start = time.perf_counter()
labels, distances = self.index.knn_query(vector.reshape(1, -1), k=k)
elapsed_ms = (time.perf_counter() - start) * 1000
if elapsed_ms > timeout_ms:
log.warning("query_slow", elapsed_ms=round(elapsed_ms, 2), threshold_ms=timeout_ms)
log.info(
"vector_query",
k=k,
elapsed_ms=round(elapsed_ms, 2),
top_score=round(float(1 - distances[0][0]), 4), # hnswlib returns cosine distance
)
similarities = [1.0 - float(d) for d in distances[0]]
return list(labels[0]), similarities
def save(self, path: str = INDEX_PATH) -> None:
self.index.save_index(path)
log.info("index_saved", path=path, element_count=self.index.element_count)
def load(self, path: str = INDEX_PATH) -> None:
self.index.load_index(path, max_elements=MAX_ELEMENTS)
log.info("index_loaded", path=path, element_count=self.index.element_count)
How this code works
This Python code defines a VectorIndex class, designed for efficiently storing and retrieving similar vector embeddings. It serves as a rapid lookup system within a lesson on vector similarity search, specifically implementing an Approximate Nearest Neighbors (ANN) search using hnswlib. This allows applications to quickly find the most relevant items (represented by vectors) from a large collection based on their "cosine" similarity.
The __init__ method sets up the hnswlib.Index with parameters like the embedding DIMension and HNSW-specific settings (M, EF_CONSTRUCTION, EF_SEARCH) that tune the index's speed and accuracy. The add_batch method ingests multiple vectors along with their ids, critically checking for "zero-norm" vectors that would cause issues with cosine similarity. When the query method searches for similar vectors, it explicitly normalizes the input vector to unit length using vector = vector / norm. This is a subtle but important step; while hnswlib is configured for "cosine" space, ensuring unit-length vectors guarantees the most accurate cosine distance (which is converted back to similarity) results. The @retry decorator from tenacity adds robustness to queries, handling potential transient errors.
Practice & master
Try the exercise, check your understanding, then mark this lesson mastered to track your path to pro.
Exercise
Build a tiny in-memory semantic search function. Given a list of 10 hard-coded sentences and a query string, embed all of them using the OpenAI embeddings API (text-embedding-3-small), then return the top-3 most similar sentences using cosine similarity computed with numpy. Print each result with its similarity score.
# openai 1.x, numpy 1.26
import os
import numpy as np
from openai import OpenAI
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
DOCS = [
"How do I reset my password?",
"What are your business hours?",
"How do I cancel my subscription?",
"Where can I find my invoice?",
"What payment methods do you accept?",
"How do I contact support?",
"Can I change my email address?",
"Is there a free trial available?",
"How do I download my data?",
"What is your refund policy?",
]
QUERY = "I want to stop paying for my plan"
def embed(texts: list[str]) -> np.ndarray:
# TODO: call client.embeddings.create with model="text-embedding-3-small"
# return a 2D numpy array of shape (len(texts), dim)
pass
def cosine_similarity_matrix(query_vec: np.ndarray, doc_vecs: np.ndarray) -> np.ndarray:
# TODO: compute cosine similarity between query_vec and each row of doc_vecs
# return a 1D array of similarity scores
pass
def search(query: str, docs: list[str], top_k: int = 3) -> list[tuple[str, float]]:
# TODO: embed query and docs, compute similarities, return top_k (doc, score) pairs
pass
if __name__ == "__main__":
results = search(QUERY, DOCS)
for doc, score in results:
print(f"{score:.4f} {doc}")
Quick check
You have unit-length (L2-normalized) vectors. Which similarity function gives the same ranking as cosine similarity but with less computation?
Why does cosine similarity often outperform dot product for comparing a short query against long documents when vectors are NOT normalized?
An ANN index built with HNSW returns recall@10 of 0.82 on your test set. What is the most direct lever to improve recall without rebuilding from scratch?