The mental model for LLM-as-a-judge is simple: evaluation is itself a text task. A human reviewer reads a question, reads an answer, compares it to some criteria in their head, and returns a verdict. That is exactly what you are asking the judge model to do. The critical insight is that generating a good answer and evaluating one are different skills. Evaluation requires reading comprehension, rubric-following, and calibration against a standard, not open-ended generation. Strong instruction-tuned models with large context windows happen to be very good at this, which is why GPT-4 class models consistently outperform smaller models as judges even when those smaller models are the SUT.
The most important engineering decision is your judge prompt. It needs four things: the original question or context, the SUT's response, an explicit and unambiguous rubric, and an output format you can parse reliably. "Rate this response" is not a rubric. A rubric looks like: "Score 1 if the response contains factual errors or is irrelevant. Score 3 if it is mostly correct but missing key details. Score 5 if it is accurate, complete, and concise." Anchor descriptions at each point on the scale. Vague criteria produce high variance scores that are useless for regression detection. Use temperature=0 on the judge to make scores deterministic, and use JSON mode or function calling to guarantee parseable output. If you ask for a free-text verdict, you will spend more time parsing than evaluating.
A real-world scenario: you maintain a customer support bot for a SaaS product. The team wants to change the system prompt to make responses more concise. Before shipping, you run your eval dataset of 500 question-answer pairs through both the old and new prompt. For each pair you call the judge with a rubric covering accuracy, conciseness, and tone. The judge returns a score for each dimension. You aggregate: new prompt scores 4.1 vs 4.3 on accuracy but 4.6 vs 3.8 on conciseness. You ship, knowing the tradeoff explicitly. Without automated eval, this comparison would require hours of human review. With it, the entire pipeline runs in under five minutes.
The main tradeoffs versus alternatives: human evaluation is the gold standard for ambiguous, high-stakes tasks (legal, medical) but does not scale and has its own inter-annotator variance. Rule-based metrics like ROUGE or BLEU are cheap and deterministic but completely miss semantic quality, hallucinations, and tone. LLM-as-a-judge sits between them: semantic, scalable, and auditable, but dependent on the judge model's own biases and capabilities. A known failure mode is positional bias: if you present two responses (A/B), many models prefer whichever comes first. Mitigate this by always randomizing order and averaging both orderings. Another failure mode is verbosity bias: judges often prefer longer answers even when a short answer is correct. Counter this by explicitly stating in your rubric that length alone does not increase the score.
At scale, the architecture changes. At 10 users, you can afford to call GPT-4o for every eval run. At 10k users with a daily eval suite of 5,000 examples, you are looking at meaningful API costs and latency. At this level, profile which rubric dimensions are most predictive of real-world quality and run a cheaper judge model (GPT-4o-mini, Haiku) for those dimensions, reserving the expensive model for ambiguous cases flagged by the cheap model. At 10M users, you likely need a dedicated fine-tuned judge model hosted internally. You can bootstrap this by distilling GPT-4o judge labels into a smaller model: collect 10,000 judge verdicts from GPT-4o, fine-tune Llama 3 8B on them, and validate that your fine-tuned judge agrees with GPT-4o on a held-out set above some agreement threshold (Cohen's kappa > 0.7 is a reasonable bar). This cuts per-evaluation cost by 10-50x.
Cost and latency implications are real. A judge call typically consumes 300-600 input tokens (question + response + rubric) plus 50-100 output tokens. At current GPT-4o pricing, a 1,000-example eval suite costs roughly $1-3, but that figure changes frequently so always check the provider's pricing page. Latency per call is 1-3 seconds. Running 1,000 calls sequentially takes 15-45 minutes. Run them concurrently with asyncio and a semaphore to cap at 20-50 concurrent requests, and the same suite finishes in under two minutes. This is the difference between an eval suite you actually run before every deploy and one that gathers dust.
Key Takeaways
- Structure your judge prompt with explicit criteria and a fixed score scale to get consistent, parseable outputs.
- Use a stronger model as judge than the SUT; judging is harder than answering.
- Always log the judge's reasoning, not just the score, so you can audit and improve the rubric.
- Calibrate judge scores against human labels on a small sample before trusting them at scale.
Pro tips
- Run the same example through your judge twice with different prompt orderings (swap question and response position) and average the scores. Single-pass judging has measurable positional bias that double-pass averaging almost eliminates.
- Before trusting your judge at scale, manually label 50-100 examples and compute Cohen's kappa or Spearman correlation between judge scores and human scores. Anything below 0.6 correlation means your rubric needs more anchor descriptions, not a better model.
- Store the full judge response, including the reasoning field, in your eval database. When you debug a score that looks wrong, the reason field almost always tells you whether the rubric is ambiguous or the SUT actually failed. Without it you are debugging blind.
- Use a separate, dedicated system prompt for your judge that says explicitly 'You are an evaluation assistant. Do not attempt to answer the question yourself.' Without this, some models will try to be helpful and rewrite the answer instead of scoring it.
Common pitfalls
- Mistake: Asking the judge to score multiple dimensions in a single free-text response. Fix: Request one JSON object with named fields per dimension so you can parse and aggregate each independently.
- Mistake: Using the same model as both SUT and judge. Fix: Use a stronger or at least different model as judge; same-model self-evaluation inflates scores and misses systematic errors.
- Mistake: Never validating judge scores against human labels. Fix: Spot-check 50 examples per rubric version and compute correlation; recalibrate rubric anchors if agreement is below 0.6.
- Mistake: Running the eval suite sequentially, making it too slow to use in CI. Fix: Use asyncio with a semaphore to cap concurrency; 1,000 calls go from 30 minutes to under 2 minutes.
When to use LLM-as-a-judge vs alternative eval methods
| Option | Use when | Avoid when |
|---|---|---|
| LLM-as-a-judge | You need semantic quality signals (helpfulness, coherence, accuracy) at scale with no reference answer required. | The task has an exact correct answer (math, code execution, factual lookup) where deterministic checks suffice. |
| Rule-based metrics (ROUGE, BLEU, exact match) | You have a gold-standard reference and need zero-cost, fully deterministic scoring that runs offline. | The response space is open-ended; paraphrased correct answers will score poorly against a fixed reference. |
| Human evaluation | You are making a high-stakes launch decision, establishing ground truth labels, or auditing your judge's calibration. | You need daily eval runs or fast iteration; human review does not scale to hundreds of examples per deploy. |
| Fine-tuned judge model | You have a large, domain-specific eval dataset and per-call API costs are prohibitive at your eval volume. | You are early in development; the upfront labeling cost (1,000+ examples) is not justified until the rubric is stable. |
Code Example
# openai>=1.30.0
import openai, json
client = openai.OpenAI()
JUDGE_PROMPT = """
You are an evaluation assistant. Score the RESPONSE on a scale of 1-5 for helpfulness.
Return JSON: {{"score": <int>, "reason": "<one sentence>"}}
QUESTION: {question}
RESPONSE: {response}
"""
def judge(question: str, response: str) -> dict:
result = client.chat.completions.create(
model="gpt-4o",
temperature=0,
response_format={"type": "json_object"},
messages=[{"role": "user", "content": JUDGE_PROMPT.format(
question=question, response=response
)}],
)
return json.loads(result.choices[0].message.content)
verdict = judge("What is RAG?", "RAG stands for Retrieval-Augmented Generation.")
print(verdict) # {"score": 3, "reason": "Correct but lacks detail."}How this code works
This code demonstrates how to use an LLM (specifically OpenAI's gpt-4o) as an automated judge to evaluate the helpfulness of another LLM's answer. The primary job is to automatically generate a score and a reason for a given response to a question, replacing manual human review. The process begins with JUDGE_PROMPT, which serves as the instruction set for the judge LLM. This prompt clearly outlines the evaluation criteria: score the RESPONSE on a 1-5 scale for helpfulness, and importantly, specifies that the output must be a JSON object containing a score and a reason.
The judge function orchestrates this evaluation. It takes the specific question and response to be evaluated and sends them to the gpt-4o model via client.chat.completions.create. A crucial detail here is the response_format={"type": "json_object"} option. This explicitly tells the judge LLM to generate its output in a strict JSON format, which is vital because it prevents common issues where LLMs might sometimes produce malformed JSON or include extra conversational text. This ensures the subsequent json.loads call can reliably convert the judge's output into a Python dictionary, making the score and reason easily accessible for further processing.
Production-grade example
Adds async concurrency, exponential backoff on rate limits, per-call timeouts, token logging, and graceful degradation on timeout.
# openai>=1.30.0, tenacity>=8.2
import asyncio, json, logging, os, time
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
import openai
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger(__name__)
client = openai.AsyncOpenAI(api_key=os.environ["OPENAI_API_KEY"])
JUDGE_PROMPT = """
Evaluate the RESPONSE against the QUESTION using this rubric:
1 = factually wrong or off-topic
3 = mostly correct, missing key details
5 = accurate, complete, appropriately concise
Return JSON only: {{"score": <1|2|3|4|5>, "reason": "<one sentence>"}}
QUESTION: {question}
RESPONSE: {response}
"""
@retry(
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=1, min=2, max=30),
retry=retry_if_exception_type((openai.RateLimitError, openai.APIConnectionError)),
)
async def judge_single(question: str, response: str, example_id: str) -> dict:
t0 = time.monotonic()
try:
result = await asyncio.wait_for(
client.chat.completions.create(
model="gpt-4o",
temperature=0,
max_tokens=150,
response_format={"type": "json_object"},
messages=[{"role": "user", "content": JUDGE_PROMPT.format(
question=question, response=response
)}],
),
timeout=20.0,
)
except asyncio.TimeoutError:
log.warning("judge_timeout example_id=%s", example_id)
return {"example_id": example_id, "score": None, "reason": "timeout", "error": True}
latency_ms = int((time.monotonic() - t0) * 1000)
usage = result.usage
log.info(
"judge_ok example_id=%s score_tokens=%d latency_ms=%d prompt_tokens=%d completion_tokens=%d",
example_id, usage.total_tokens, latency_ms, usage.prompt_tokens, usage.completion_tokens,
)
payload = json.loads(result.choices[0].message.content)
return {"example_id": example_id, **payload, "latency_ms": latency_ms, "error": False}
async def run_eval(examples: list[dict], concurrency: int = 20) -> list[dict]:
sem = asyncio.Semaphore(concurrency)
async def bounded(ex):
async with sem:
return await judge_single(ex["question"], ex["response"], ex["id"])
return await asyncio.gather(*[bounded(e) for e in examples])
if __name__ == "__main__":
sample = [
{"id": "ex1", "question": "What is RAG?", "response": "Retrieval-Augmented Generation combines retrieval with LLM generation."},
{"id": "ex2", "question": "What is RAG?", "response": "RAG is a type of sandwich."},
]
results = asyncio.run(run_eval(sample))
for r in results:
print(r)How this code works
This code automates the evaluation of LLM responses using another LLM as a judge, a core concept in LLM Evaluation & Testing. It processes a list of examples, each containing a question and a response, and sends them to an OpenAI gpt-4o model. The judge model scores the response against the question based on a predefined JUDGE_PROMPT rubric and returns a score and reason in JSON, effectively providing automated feedback on the quality of responses. The run_eval function manages these evaluations concurrently for efficiency.
At its heart, the judge_single function makes the actual call to the OpenAI API using client.chat.completions.create. It's enhanced with the @retry decorator from tenacity, which automatically re-attempts the API request up to three times if it encounters RateLimitError or APIConnectionError, making the evaluation robust against temporary network or API issues. A subtle but critical detail for beginners is the response_format={"type": "json_object"} parameter inside the create call. This setting explicitly tells the OpenAI model to produce a JSON object, significantly increasing the likelihood that json.loads will successfully parse the judge's output without errors, ensuring the evaluation results are reliably structured. The asyncio.Semaphore in run_eval helps control the concurrency of simultaneous judge calls to prevent overwhelming the API.
Practice & master
Try the exercise, check your understanding, then mark this lesson mastered to track your path to pro.
Exercise
Build a simple eval pipeline that scores five customer support responses on a 1-5 helpfulness scale using an LLM judge. The pipeline should return a summary dict with the mean score, the lowest-scoring example ID, and the judge's reason for that low score. Use GPT-4o-mini as the judge to keep costs low.
# openai>=1.30.0
import openai, json, os
client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])
EXAMPLES = [
{"id": "e1", "question": "How do I reset my password?", "response": "Click Forgot Password on the login page."},
{"id": "e2", "question": "Why was I charged twice?", "response": "I don't know, contact someone else."},
{"id": "e3", "question": "Can I export my data?", "response": "Yes, go to Settings > Data > Export and choose CSV or JSON."},
{"id": "e4", "question": "Is there a mobile app?", "response": "Yes."},
{"id": "e5", "question": "How do I cancel my subscription?", "response": "Go to Billing > Cancel and follow the prompts. You keep access until the period ends."},
]
JUDGE_PROMPT = """
# TODO: Write a rubric prompt that returns JSON with 'score' (1-5) and 'reason'
"""
def judge_response(example: dict) -> dict:
# TODO: Call the judge model and parse the JSON response
pass
def run_eval(examples: list) -> dict:
results = [judge_response(e) for e in examples]
# TODO: Compute mean score, find lowest-scoring example, return summary
pass
if __name__ == "__main__":
summary = run_eval(EXAMPLES)
print(summary)Quick check
You run your LLM judge and notice it consistently gives higher scores to longer responses regardless of accuracy. What is the most targeted fix?
Why should you use temperature=0 when calling the judge model, but not necessarily when calling the SUT?
You want to validate your LLM judge before relying on it for CI gating. What is the correct first step?