The mental model for an embedding model is a function that maps a string of arbitrary length into a fixed-size point in high-dimensional space. Two strings with similar meanings end up close together; unrelated strings end up far apart. The model is a neural network trained (typically with contrastive learning) on pairs of sentences judged similar or dissimilar. It never generates text, it only encodes. This is why embedding models are fundamentally different from GPT-style completion models even when they come from the same provider. They are optimized for representation quality, not generation fluency.
OpenAI's current recommended models are text-embedding-3-small (1536 dimensions, ~$0.02 per million tokens as of early 2025, verify current pricing) and text-embedding-3-large (3072 dimensions, roughly 5x the cost). Both support dimension reduction via the dimensions parameter, which truncates and renormalizes the vector. You might use 512 dimensions in a Pinecone free tier to stay within storage limits, then promote to 1536 if your recall metrics suffer. The predecessor, text-embedding-ada-002, is still functional but newer models outperform it on MTEB benchmarks and cost less per token. Unless you have an existing ada-002 index you cannot afford to rebuild, there is no reason to start a new project on it.
Cohere's Embed v3 family introduces a meaningful differentiation: input_type. You pass "search_document" when embedding corpus chunks at index time and "search_query" when embedding a user query at retrieval time. This asymmetric embedding approach consistently improves retrieval precision because the model learns that a query and its relevant document have different linguistic structures. embed-english-v3.0 targets English-only corpora; embed-multilingual-v3.0 handles 100+ languages with competitive performance on cross-lingual tasks. If your application needs to match an English query against Spanish or Japanese documents, Cohere's multilingual model is usually a better starting point than trying to force everything through an English-primary model.
Open-source alternatives run on your own hardware or a managed GPU instance. The Sentence Transformers library gives you a clean interface to hundreds of models on Hugging Face. Practically useful today: BAAI/bge-large-en-v1.5 (1024 dims, strong on MTEB English), intfloat/e5-large-v2 (1024 dims, similar performance tier), and all-MiniLM-L6-v2 (384 dims, very fast, lower quality). BGE models also support instruction prefixes similar to Cohere's input_type: prepend "Represent this sentence for searching relevant passages:" to documents and "Represent this sentence for searching queries:" to queries. A real production scenario: a legal tech startup with 50M contract clauses cannot afford $1,000+ per full re-embed cycle, so they host bge-large on a single A10G instance, pay ~$0.80/hour, and amortize that over continuous ingestion. The operational cost of managing the endpoint beats the per-token cost at their volume.
Tradeoffs at different scales: at 10 users, just use the OpenAI API, the simplicity is worth more than the cost savings. At 10,000 users with moderate query volume, token costs become visible but API latency (typically 100-400ms per batch) is usually acceptable. At 10 million users or with a continuous ingestion pipeline processing millions of documents per day, you will feel both the cost and the latency ceiling of external APIs. Batch size matters: OpenAI accepts up to 2048 inputs per request; Cohere accepts up to 96; Sentence Transformers batch size is limited by your GPU VRAM. Mismatching batch sizes to API limits is a common source of throughput bottlenecks. Dimension count directly affects your vector database storage and ANN index memory footprint, which translates to infrastructure cost. Cutting from 3072 to 768 dimensions might cost a few recall points on your benchmarks but halve your Pinecone pod size.
Key Takeaways
- Re-embedding a corpus when switching models is expensive; benchmark models against your data before committing.
- OpenAI text-embedding-3-small covers most use cases at lower cost than text-embedding-3-large.
- Use Cohere embed-multilingual-v3.0 when your corpus spans multiple languages.
- Self-hosted BGE or E5 models eliminate per-call costs but add infrastructure and latency management overhead.
Pro tips
- Always benchmark on your own data, not just MTEB leaderboard scores. A model ranked 3rd on MTEB might outrank the top model on your specific domain because the benchmark corpus does not match your text distribution.
- When using Cohere Embed v3 or BGE with instruction prefixes, embed your queries at search time with the query prefix and your documents at index time with the document prefix. Mixing these up is silent and causes measurably worse recall.
- For open-source models, the first inference call after loading takes 2-5x longer because of JIT compilation and GPU warm-up. Always send a dummy warm-up batch before your first real request in a long-running service.
- Store the model name and dimension count as metadata next to every vector you persist. When you upgrade the embedding model six months later, you need to know which index entries need to be re-embedded and which do not.
Common pitfalls
- Mistake: Embedding very long documents as a single input, ignoring the model's token limit. Fix: Check each model's max token limit (e.g., 8191 for text-embedding-3-small) and chunk documents before embedding.
- Mistake: Comparing cosine similarity scores across different embedding models to rank retrieval quality. Fix: Scores are not comparable across models; always evaluate recall@k on a fixed test set with ground truth.
- Mistake: Using the same text-embedding-3 dimensions parameter for all environments to save config complexity. Fix: Use lower dimensions in dev/staging to reduce cost, but validate on full dimensions before a production rollout.
- Mistake: Sending one text per API call in a loop instead of batching. Fix: Batch up to the model's limit per request; single-text calls waste round-trip latency and exhaust rate limits faster.
When to use OpenAI vs Cohere vs open-source embedding models
| Option | Use when | Avoid when |
|---|---|---|
| OpenAI text-embedding-3-small | English-primary corpus, moderate volume, want minimal ops, dimension reduction useful for DB cost control. | Budget is very tight at high volume or you need cross-lingual matching beyond basic English. |
| OpenAI text-embedding-3-large | Maximum retrieval quality matters more than cost; running a premium product where recall failures are expensive. | Cost per embedding is a primary constraint or you have not verified the quality lift justifies ~5x price increase. |
| Cohere embed-multilingual-v3.0 | Corpus or queries span multiple languages; want asymmetric input_type embeddings without self-hosting. | English-only use case where OpenAI or a self-hosted model is already meeting recall targets. |
| BGE / E5 via Sentence Transformers | High volume makes API costs significant; data privacy requirements prohibit sending text to external APIs; need offline operation. | Team lacks GPU infrastructure experience; you are in early prototype phase and want to move fast without ops work. |
| all-MiniLM-L6-v2 | Latency is the primary constraint and you can tolerate lower recall; edge deployment or very resource-constrained environments. | Retrieval quality matters; this model is fast but measurably worse than BGE-large or E5-large on most benchmarks. |
Code Example
# openai>=1.0.0, cohere>=5.0.0, sentence-transformers>=2.7.0
import openai
client = openai.OpenAI() # reads OPENAI_API_KEY from env
def embed_openai(texts: list[str], model: str = "text-embedding-3-small") -> list[list[float]]:
response = client.embeddings.create(input=texts, model=model)
return [item.embedding for item in response.data]
docs = ["Retrieval-augmented generation grounds LLMs in facts.", "Vector search finds semantically similar documents."]
vectors = embed_openai(docs)
print(f"Model: text-embedding-3-small | Dimensions: {len(vectors[0])}") # 1536How this code works
This code demonstrates how to convert human language into numerical representations called "embeddings" using OpenAI's models. These embeddings capture the meaning of text, allowing computers to understand semantic similarity, which is a core concept for vector search and other AI applications. The process starts by bringing in the openai library and creating an openai.OpenAI() client object. A crucial detail for beginners is that this client automatically reads the OPENAI_API_KEY from the system's environment variables, keeping sensitive credentials out of the code itself.
The embed_openai function is where the magic happens. It takes a list of text strings and an optional model name, then calls client.embeddings.create to send the text to OpenAI. The API response contains the embeddings, which are then neatly extracted using a list comprehension [item.embedding for item in response.data]. When embed_openai is called with the example docs, a subtle but important point is that the model parameter has a default value of "text-embedding-3-small". This means if no model is explicitly provided during the function call, this well-balanced model will be used automatically. The final print statement confirms the model used and the dimensionality (length) of the resulting numerical vectors.
Production-grade example
Retries on transient errors, per-batch token logging, configurable dimensions, timeout, and structured log context.
# openai>=1.0.0, tenacity>=8.2.0
import os
import logging
import time
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
import openai
from openai import RateLimitError, APITimeoutError, APIConnectionError
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
logger = logging.getLogger(__name__)
client = openai.OpenAI(
api_key=os.environ["OPENAI_API_KEY"],
timeout=30.0,
)
EMBED_MODEL = "text-embedding-3-small"
EMBED_DIMS = 1536
BATCH_SIZE = 512 # stay well under the 2048-input limit to avoid large payload rejections
@retry(
retry=retry_if_exception_type((RateLimitError, APITimeoutError, APIConnectionError)),
wait=wait_exponential(multiplier=1, min=2, max=60),
stop=stop_after_attempt(5),
reraise=True,
)
def _embed_batch(texts: list[str], model: str, dimensions: int) -> list[list[float]]:
response = client.embeddings.create(input=texts, model=model, dimensions=dimensions)
tokens_used = response.usage.total_tokens
logger.info("embedded batch", extra={"batch_size": len(texts), "tokens": tokens_used, "model": model})
return [item.embedding for item in response.data]
def embed_texts(
texts: list[str],
model: str = EMBED_MODEL,
dimensions: int = EMBED_DIMS,
batch_size: int = BATCH_SIZE,
) -> list[list[float]]:
if not texts:
return []
results: list[list[float]] = []
total_start = time.monotonic()
for i in range(0, len(texts), batch_size):
chunk = texts[i : i + batch_size]
try:
batch_vectors = _embed_batch(chunk, model=model, dimensions=dimensions)
results.extend(batch_vectors)
except Exception as exc:
logger.error("embedding batch failed after retries", extra={"batch_start": i, "error": str(exc)})
raise
elapsed = time.monotonic() - total_start
logger.info("embedding complete", extra={"total_texts": len(texts), "elapsed_s": round(elapsed, 2)})
return resultsHow this code works
This Python code is designed to efficiently generate numerical "embeddings" for a list of text inputs using OpenAI's AI models, a fundamental process for tasks like semantic search or text comparison. It intelligently handles communication with the OpenAI API, ensuring reliability even when faced with common network or service limitations.
The code initializes an openai.OpenAI client, fetching the OPENAI_API_KEY securely from environment variables, and defines constants like EMBED_MODEL and BATCH_SIZE. The critical work is performed by the _embed_batch function, which makes the actual client.embeddings.create API call. A key robustness feature here is the @retry decorator from tenacity; it automatically retries the API call if transient errors like RateLimitError or network connection issues occur, using an exponential backoff (wait_exponential) to prevent overwhelming the API, a subtle but vital detail for production systems. The main embed_texts function orchestrates this by dividing the input texts into smaller batch_size chunks, then iteratively calls _embed_batch for each chunk to stay within API limits and collects all the resulting embedding vectors, logging progress along the way.
Practice & master
Try the exercise, check your understanding, then mark this lesson mastered to track your path to pro.
Exercise
Build a function that embeds a list of documents using OpenAI text-embedding-3-small, then embeds a query, and returns the top-2 most similar documents by cosine similarity using only the standard library and openai. Print the ranked documents and their similarity scores.
# openai>=1.0.0
import os
import math
import openai
client = openai.OpenAI() # reads OPENAI_API_KEY from env
DOCS = [
"The Eiffel Tower is located in Paris, France.",
"Python is a popular programming language for data science.",
"Neural networks learn representations from data.",
"The Louvre Museum houses the Mona Lisa painting.",
]
QUERY = "famous landmarks in France"
def cosine_similarity(a: list[float], b: list[float]) -> float:
# TODO: implement dot product divided by product of magnitudes
pass
def embed(texts: list[str]) -> list[list[float]]:
# TODO: call client.embeddings.create with model="text-embedding-3-small"
pass
def top_k_docs(query: str, docs: list[str], k: int = 2) -> list[tuple[str, float]]:
# TODO: embed docs and query, compute similarities, return top-k (doc, score) pairs
pass
if __name__ == "__main__":
results = top_k_docs(QUERY, DOCS)
for doc, score in results:
print(f"{score:.4f} {doc}")Quick check
You switch from text-embedding-ada-002 to text-embedding-3-small on an existing vector index without re-embedding. What happens?
Cohere's embed-multilingual-v3.0 requires you to specify input_type. Which pair is correct for a RAG pipeline?
A team reduces text-embedding-3-large output from 3072 to 512 dimensions using the API's dimensions parameter. What is the main risk?