AI & ML

Vector Embeddings

Numerical representations of text, images, or other data in high-dimensional space, where semantically similar items are positioned close together.

Vector embeddings are dense numerical representations of data (text, images, audio) in high-dimensional space. They capture semantic meaning, so items with similar meanings have similar vector representations.

For text, an embedding model converts a sentence like "How do I deploy to production?" into a vector of 768-3072 floating-point numbers. Similar questions like "What's the deployment process?" would produce vectors that are close in this space, even though they use different words.

Embeddings are fundamental to modern AI applications. They power semantic search (finding relevant results by meaning, not keywords), recommendation systems, clustering, anomaly detection, and RAG pipelines.

Popular embedding models include OpenAI's text-embedding-3, Cohere's embed-v3, and open-source options like sentence-transformers. The choice of embedding model affects the quality of downstream tasks and the dimensionality of the vectors stored.

How it works

An embedding model is a neural network trained so that inputs used in similar contexts end up at nearby points. Word2vec (Mikolov et al., 2013) popularised this for single words and learned high-quality word vectors from a 1.6-billion-word dataset in under a day; current models embed whole sentences and passages.

The output is a fixed-length list of numbers no matter how long the input is. OpenAI's documentation gives 1,536 dimensions by default for text-embedding-3-small and 3,072 for text-embedding-3-large, and a dimensions parameter to shorten them; it reports that a text-embedding-3-large vector cut to 256 dimensions still outperforms the older text-embedding-ada-002 at 1,536.

Closeness is measured with a similarity function, most often cosine similarity: the cosine of the angle between two vectors, from -1 to 1. When vectors are normalised to length 1, as OpenAI's are, cosine similarity equals the dot product and gives the same ranking as Euclidean distance.

Vectors from different models live in different spaces and cannot be compared with each other. Switching embedding models means re-embedding the whole corpus.

Example

Cosine similarity by hand with toy three-dimensional vectors. Real embeddings have hundreds or thousands of dimensions, but the arithmetic is the same.

const dot = (u, v) => u.reduce((sum, x, i) => sum + x * v[i], 0);
const norm = (u) => Math.sqrt(dot(u, u));
const cosine = (u, v) => dot(u, v) / (norm(u) * norm(v));

const deploy    = [0.8, 0.6, 0.0]; // "deploy to production"
const release   = [0.6, 0.8, 0.0]; // "ship a release"
const sourdough = [0.0, 0.0, 1.0]; // "bake sourdough"

cosine(deploy, release);   // ≈ 0.96  (0.8×0.6 + 0.6×0.8)
cosine(deploy, sourdough); // 0       (no shared direction)

Both deploy vectors already have length 1, so the cosine is just the dot product: 0.48 + 0.48 = 0.96. The sourdough vector points in a direction neither of them uses, so its similarity is 0. Semantic search is this calculation repeated against every stored vector, or against an index that finds the top results without checking them all.

Common confusion

What is the difference between embeddings and tokens?

A token ID is an integer index into a model's vocabulary; its numeric value carries no meaning. An embedding is a learned vector of real numbers in which distances do carry meaning. Inside a language model each token ID is looked up in an embedding table, while an embedding API returns one vector representing an entire piece of text.

What is the difference between an embedding and a vector database?

The embedding is the data: a vector produced by a model. A vector database is the system that stores many of those vectors and indexes them so the nearest ones to a query can be found quickly. Small projects can keep embeddings in memory or in an ordinary database without a dedicated vector store.

Embedding Space Explorer Explore word embeddings in 2D and try vector arithmetic.
AI Tokenizer Visualizer See the token IDs that come before embeddings.
Embedding Model Prices Compare what it costs to embed a corpus with different models.
Semantic Search The most common use of text embeddings.
Vector Database Where embeddings are stored and searched.