Treating prompts as code means applying every discipline you already use for source code: the prompt text lives in a repository, changes go through pull requests with descriptions explaining why a change was made (not just what), and every deployed version carries an identifier that can be matched back to a specific commit or registry entry. The most common failure pattern is the opposite: a prompt lives in a Python string inside an application file, gets edited directly in production config, and no one has a clear record of what changed when. When quality drops a week later, you are debugging blind.
The mental model for a prompt registry is simple: name plus version equals a stable key. "summarize:v1" and "summarize:v2" are different artifacts. Your application code references the version explicitly, or it asks a routing layer which version to use for a given request. This decoupling means you can swap the prompt under a feature without touching your service logic. In Git terms, this is just storing your prompts in a /prompts directory with YAML or JSON files and committing them like any other source file. Platforms like LangSmith, Humanloop, Weights and Biases Prompts, and PromptLayer layer a UI and API on top of this concept, but the Git-based baseline is free and gets you 80% of the value immediately.
A/B testing a prompt is structurally identical to A/B testing a UI change: you need a traffic splitter, a way to log which variant a request got, and a metric you care about. The tricky part with LLM outputs is that your primary metric is often quality, and quality is not a number you get for free. You have three practical options. First, use an LLM-as-judge: send the output to a grader model (GPT-4o or a fine-tuned classifier) with a rubric and get back a 1-5 score. Second, use downstream behavioral signals: did the user accept the generated draft, or did they rewrite it? Did the support ticket get resolved without a follow-up? Third, use human evaluators for high-stakes cases, sampling a small percentage of responses for manual scoring. Most production teams combine all three. LLM-as-judge is cheap and fast, behavioral signals are the ground truth but arrive with lag, and human eval validates the judge.
A real-world scenario: you run a customer-facing summarization feature. Your current prompt (v1) produces summaries that users sometimes complain are too verbose. You write v2 with an explicit length constraint and stronger instruction framing. Before pushing v2 to 100% of traffic, you configure your feature flag system (LaunchDarkly, Unleash, or a simple Redis-backed hash) to send 10% of requests to v2. You log the prompt version, request ID, latency, token count, and LLM-as-judge quality score on every call. After 48 hours and roughly 2,000 requests in each bucket, you run a t-test on quality scores and a Mann-Whitney U test on latency distributions. If v2 wins on quality with p < 0.05 and does not regress on latency or cost, you promote it. That whole sequence is a week of work the first time and 20 minutes the second time.
Tradeoffs versus alternatives: some teams skip formal A/B testing and use shadow mode instead. Shadow mode runs the new prompt in parallel but never shows its output to users. You get cost and latency data but no behavioral signal. It is useful for catching catastrophic regressions before they hit users, but it cannot tell you whether v2 is better than v1 from a user perspective. The other common alternative is offline evaluation: run both prompts on a benchmark dataset and compare scores before deploying either. Offline eval is fast and cheap but often misses distribution shift between your benchmark and real traffic. A production A/B test is slower but more trustworthy. In practice, you want both: offline eval to catch obvious regressions early, and A/B testing to confirm improvements on real traffic.
At scale, the operational concerns shift. At 10 users, Git plus a spreadsheet is fine. At 10,000 requests per day, you need structured logging with prompt version in every log line and a dashboard that shows metric breakdowns by version. At 10 million requests per day, you need a full prompt registry service, automated statistical significance calculations, guardrails on traffic allocation (so a bad prompt cannot stay at 50% for a week while draining quality), and rollback automation triggered by metric degradation. Cost also becomes a meaningful variable at scale: v2 might use 20% more tokens for a marginal quality gain, which could represent thousands of dollars per month. The A/B test data helps you make that call with numbers rather than intuition.
Key Takeaways
- Store prompts as versioned artifacts with semantic IDs, not hardcoded strings in application code.
- A/B test prompts against measurable outcomes: quality scores, latency, cost, or task completion rates.
- Use feature flags or a prompt registry to route traffic between versions without code deploys.
- Wait for statistical significance before promoting a new prompt version; gut feel is not evidence.
Pro tips
- Hash the request ID to pick a variant, not random.random(). Hashing gives you sticky assignment, so the same user always sees the same prompt version within an experiment window. Random re-rolls on retries can contaminate your data.
- Log the exact prompt text hash alongside the version label. Version labels are human-assigned and can drift; the content hash is the ground truth. If someone edits a prompt file without bumping the version, the hash catches it.
- Set a minimum detectable effect before you start the experiment. If you need a 3-point quality improvement to justify shipping, calculate the sample size required to detect that at p < 0.05 before touching traffic. Otherwise you'll declare victory or defeat too early.
- Keep experiments short and bounded. A/B tests running longer than 2 to 3 weeks accumulate novelty effects, seasonal drift, and model version changes from your provider. Treat a long-running experiment as a signal that your metric or traffic split is wrong.
Common pitfalls
- Mistake: Changing the prompt mid-experiment to fix a small issue without starting a new experiment. Fix: Treat any prompt edit as a new version. Stop the current test, reset metrics, and restart.
- Mistake: Using the same evaluation LLM as the generation LLM as your judge. Fix: Use a different model or a purpose-built classifier as judge; the generating model will self-favor.
- Mistake: Declaring a winner after 50 samples because the chart looks good. Fix: Calculate required sample size upfront using a power analysis; use scipy.stats or an online calculator before collecting data.
- Mistake: Storing prompt versions only in application config without tying them to a git commit. Fix: Reference the git SHA or a semver tag in every log line so you can reconstruct exactly what ran in production.
When to use each prompt evaluation strategy
| Option | Use when | Avoid when |
|---|---|---|
| Git-based versioning only | Small team, low traffic, prompts change rarely, no dedicated prompt infra budget. | Multiple engineers editing prompts concurrently, or you need production metric dashboards. |
| Prompt registry service (Humanloop, LangSmith, PromptLayer) | You need a UI, non-engineers need to edit prompts, or you want built-in eval pipelines. | You have strict data residency requirements or want to avoid third-party vendor lock-in. |
| Offline evaluation on benchmark dataset | Catching obvious regressions before deployment, or during active prompt development iteration. | Your real traffic distribution differs significantly from your benchmark; signal will be misleading. |
| Live A/B test with LLM-as-judge | You need real-traffic signal on quality improvements and have enough volume for statistical significance. | Traffic is under a few hundred requests per day; you won't reach significance before the experiment drifts. |
| Shadow mode (parallel execution, no user exposure) | You want cost/latency data on a new prompt before any user exposure, or to catch crashes early. | You need behavioral signal (did the user accept the output?), which shadow mode cannot provide. |
Code Example
# langchain==0.2.x or standalone — pure Python, no framework required
import hashlib, json
def load_prompt(name: str, version: str, registry: dict) -> str:
"""Fetch a versioned prompt string from an in-memory registry."""
key = f"{name}:{version}"
if key not in registry:
raise KeyError(f"Prompt '{key}' not found in registry.")
return registry[key]
# Simulated prompt registry (in prod, back this with a DB or file store)
PROMPT_REGISTRY = {
"summarize:v1": "Summarize the following text in three bullet points:\n\n{text}",
"summarize:v2": "You are a concise assistant. Summarize the following text in exactly three bullet points, each under 20 words:\n\n{text}",
}
def prompt_id(name: str, version: str, text: str) -> str:
"""Deterministic ID for logging which prompt+input produced a given output."""
payload = json.dumps({"name": name, "version": version, "text": text})
return hashlib.sha256(payload.encode()).hexdigest()[:12]
template = load_prompt("summarize", "v2", PROMPT_REGISTRY)
filled = template.format(text="Large language models are trained on vast datasets...")
pid = prompt_id("summarize", "v2", filled)
print(f"[prompt_id={pid}] Sending to LLM:\n{filled}")How this code works
This code provides a fundamental way to manage and track different versions of AI prompts, a critical step for A/B testing and evaluating their performance. It begins by defining a PROMPT_REGISTRY, a simple dictionary that stores various prompt templates, each uniquely identified by a name:version key, such as summarize:v1 or summarize:v2. The load_prompt function then fetches a specific version of a prompt from this registry, ensuring the correct template is used and raising a KeyError if the requested version doesn't exist.
A key component is the prompt_id function, which creates a short, deterministic identifier for any given combination of a prompt's name, version, and the text input it receives. This ensures that the exact same prompt and input will always generate the identical prompt_id, crucial for consistent logging and comparing experimental results. A subtle but important detail is the use of json.dumps inside prompt_id; this standardizes the string representation of the input payload before hashing, guaranteeing that the hash is consistent even if the input parameters were structured slightly differently. The example then demonstrates loading a prompt version (summarize:v2), populating it with text, and calculating its unique prompt_id before simulating sending it to an LLM.
Production-grade example
Sticky deterministic routing, retries with backoff, token logging, and graceful degradation on API errors.
# Requires: openai>=1.0, structlog, tenacity, python-dotenv
import os, time, random, hashlib, logging
from typing import Literal
from openai import OpenAI, APITimeoutError, RateLimitError, APIStatusError
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
import structlog
from dotenv import load_dotenv
load_dotenv()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
log = structlog.get_logger()
PROMPT_REGISTRY = {
"summarize:v1": "Summarize the following text in bullet points:\n\n{text}",
"summarize:v2": "You are a concise assistant. Summarize in exactly 3 bullet points, each under 20 words:\n\n{text}",
}
def select_variant(request_id: str, traffic_split: float = 0.1) -> Literal["v1", "v2"]:
"""Deterministic, sticky assignment based on request_id hash."""
bucket = int(hashlib.md5(request_id.encode()).hexdigest(), 16) % 100
return "v2" if bucket < int(traffic_split * 100) else "v1"
@retry(
retry=retry_if_exception_type((APITimeoutError, RateLimitError)),
wait=wait_exponential(multiplier=1, min=1, max=30),
stop=stop_after_attempt(4),
)
def call_llm(prompt: str, model: str = "gpt-4o-mini", timeout: float = 15.0) -> dict:
start = time.monotonic()
response = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
timeout=timeout,
)
latency_ms = (time.monotonic() - start) * 1000
return {
"content": response.choices[0].message.content,
"prompt_tokens": response.usage.prompt_tokens,
"completion_tokens": response.usage.completion_tokens,
"latency_ms": round(latency_ms, 1),
}
def summarize(text: str, request_id: str) -> str:
variant = select_variant(request_id)
key = f"summarize:{variant}"
prompt = PROMPT_REGISTRY[key].format(text=text)
try:
result = call_llm(prompt)
log.info(
"llm_call_ok",
request_id=request_id,
prompt_version=variant,
prompt_tokens=result["prompt_tokens"],
completion_tokens=result["completion_tokens"],
latency_ms=result["latency_ms"],
# Attach cost hint; real cost calc depends on current pricing
token_total=result["prompt_tokens"] + result["completion_tokens"],
)
return result["content"]
except APIStatusError as exc:
log.error("llm_call_failed", request_id=request_id, status=exc.status_code, detail=str(exc))
return "[summarization unavailable]" # graceful degradation
if __name__ == "__main__":
import uuid
req_id = str(uuid.uuid4())
output = summarize("Large language models learn from vast amounts of text data...", req_id)
print(output)How this code works
This code demonstrates how to implement prompt versioning and A/B testing for an AI application. It defines a PROMPT_REGISTRY to store different versions of a "summarize" prompt, like summarize:v1 and summarize:v2. The core mechanism for A/B testing is the select_variant function. Given a unique request_id, it uses a cryptographic hash (hashlib.md5) to deterministically assign a prompt variant. This ensures that a specific user or request always sees the same variant ("sticky assignment"), which is vital for fair A/B test comparisons. A traffic_split parameter, for instance 0.1 for 10%, routes a controlled percentage of requests to the newer prompt version (v2), while the rest go to v1.
The call_llm function handles the actual interaction with the OpenAI API. It's adorned with a @retry decorator using tenacity, which automatically reattempts the API call if transient errors like APITimeoutError or RateLimitError occur, making the system more robust. The main summarize function orchestrates the process: it first selects a variant using select_variant, formats the chosen prompt, then calls the LLM. A subtle but important detail is the select_variant function's use of request_id for deterministic assignment, ensuring consistent user experiences across repeated requests for an A/B test. Upon completion, structlog.get_logger() is used to record structured details about the LLM call for later analysis. If the LLM call fails due to APIStatusError, it gracefully degrades by returning a fallback message.
Practice & master
Try the exercise, check your understanding, then mark this lesson mastered to track your path to pro.
Exercise
Build a minimal prompt A/B testing harness in Python. It should: (1) store two versions of a prompt in a registry dict, (2) assign a request to a variant using a deterministic hash of the request ID, (3) call the OpenAI chat completions API with the selected prompt, and (4) log the variant name, token counts, and latency to stdout as structured JSON.
import os, hashlib, json, time
from openai import OpenAI
from dotenv import load_dotenv
load_dotenv()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
# TODO 1: Define PROMPT_REGISTRY with keys "classify:v1" and "classify:v2"
# v1: simple instruction, v2: add a role and explicit output format
PROMPT_REGISTRY = {}
# TODO 2: Implement select_variant(request_id, traffic_split=0.5) -> "v1" | "v2"
# Use hashlib.md5 on the request_id to get a deterministic bucket (0-99)
def select_variant(request_id: str, traffic_split: float = 0.5) -> str:
pass
# TODO 3: Implement run_experiment(text, request_id)
# - Select variant
# - Build prompt from registry
# - Call the API (model="gpt-4o-mini")
# - Log a JSON line: {request_id, variant, prompt_tokens, completion_tokens, latency_ms}
# - Return the response text
def run_experiment(text: str, request_id: str) -> str:
pass
if __name__ == "__main__":
import uuid
for i in range(5):
req_id = str(uuid.uuid4())
result = run_experiment("The product arrived damaged and two days late.", req_id)
print("Output:", result[:80])Quick check
Why should you use a hash of the request ID rather than random.random() to assign a variant in an A/B test?
Your LLM-as-judge evaluation shows prompt v2 scoring higher than v1. Why might this result be misleading?
You run a prompt A/B test for 3 days and collect 80 samples per variant. The p-value is 0.04. Should you ship v2?