Self-consistency works because language model errors are not systematic in the same way across independent samples. When you set temperature above zero, the model explores different reasoning paths through its probability space. On a hard arithmetic or logic problem, the model might take an incorrect shortcut 30% of the time, but that shortcut rarely leads to the same wrong answer every time it fails. Correct answers, by contrast, tend to cluster. Majority voting exploits this asymmetry: the correct answer accumulates votes faster than any single incorrect answer does.
The mental model that helps most is thinking of each sample as an independent trial with some probability p of being correct. If p = 0.7 and you take a single sample, your accuracy is 70%. With 5 independent samples and majority voting (needing at least 3 votes), the probability that the majority is correct is the binomial CDF P(X >= 3) where X ~ Binomial(5, 0.7), which comes out around 84%. At 9 samples you approach 90%. The math breaks down when errors are correlated (e.g., a consistent misconception baked into the model), so self-consistency is not a substitute for prompt quality -- it amplifies whatever accuracy you already have.
A real-world scenario where this pattern earns its cost: you're building a contract clause classifier. Each contract clause gets labeled as one of ["Indemnification", "IP Assignment", "Termination", "Governing Law", "Other"]. In a quick eval, GPT-4o-mini gets 81% accuracy on a validation set with a single call. With 7 samples and majority voting, that climbs to 91% -- and crucially, you now have a confidence score (the vote fraction) you can use to route low-confidence clauses to a human reviewer. The 91% bucket is auto-processed; the low-agreement bucket gets flagged. That routing alone can justify the cost.
Tradeoffs against alternatives: prompt chaining lets you decompose a problem into sequential steps, which reduces the per-step difficulty. Self-consistency is orthogonal -- you can apply it to individual steps in a chain. Fine-tuning the model on your domain can push single-call accuracy above what self-consistency achieves at 5x cost, but requires labeled data and a deployment pipeline. Retrieval-augmented generation reduces factual errors by grounding the model, but does nothing for reasoning errors on problems where all the facts are already in the prompt. Self-consistency is your cheapest lever when you have a reasoning-heavy task and no labeled training data.
At scale, the cost and latency profile changes the calculus. At 10 users, 5x cost is trivial. At 10k users, you start profiling which questions actually need self-consistency and which don't -- low-entropy questions (ones where the model is already highly confident on a single sample) don't benefit from it and just burn tokens. At 10M users, you almost certainly need to run samples concurrently using async calls (see aidev-python-async), batch requests to hit better throughput tiers, and implement adaptive N -- start with 3 samples, check agreement, only fire 2 more if the first batch is split. You also need to store the per-question agreement rate in your observability layer so you can alert when aggregate confidence drops, which is an early signal that a model update or a prompt regression degraded accuracy.
Key Takeaways
- Sample the same prompt N times with non-zero temperature, then aggregate to reduce variance.
- Use majority voting for categorical outputs; use median or trimmed mean for numeric outputs.
- Self-consistency multiplies token cost by N, so profile before applying it everywhere.
- Track per-question confidence (agreement rate) as an observable signal, not just the final answer.
Pro tips
- Don't aggregate raw text strings with Counter unless you've normalized them first. Strip whitespace, lowercase, and optionally extract just the labeled field. Two identical answers that differ by trailing punctuation will split your vote and tank your confidence scores silently.
- Temperature between 0.6 and 0.9 is the sweet spot for most self-consistency use cases. Too low and all samples converge to the same (possibly wrong) answer -- you've added cost without diversity. Too high and answers become noisy enough that voting stops working.
- Adaptive N saves real money in production. Run 3 samples first. If agreement is 3/3 or the confidence already exceeds your threshold, short-circuit and skip the remaining calls. On typical classification tasks this cuts your average N from 7 down to roughly 3.5.
- Log the full distribution of answers per question, not just the winner. When a model update ships and your aggregate confidence drops from 0.85 to 0.71 across 500 requests, that histogram tells you exactly which classes are getting confused -- far more useful than an accuracy regression alert alone.
Common pitfalls
- Mistake: Using temperature=0 and expecting diversity across samples. Fix: Self-consistency requires stochastic sampling (temperature > 0) or top-p < 1.0 to generate meaningfully different reasoning paths.
- Mistake: Applying self-consistency to every call uniformly. Fix: Profile your pipeline and apply it only to high-stakes or low-confidence decisions; deterministic lookups and simple formatting tasks don't benefit.
- Mistake: Majority-voting raw LLM answers that contain chain-of-thought reasoning plus a final answer. Fix: Extract only the final answer token before voting, not the full reasoning text, or votes will almost never match.
- Mistake: Ignoring partial failures when some of the N async calls error out. Fix: Track how many samples succeeded; if fewer than ceil(N/2)+1 come back, you lack a reliable majority and should fall back or flag for review.
When to use self-consistency vs other reliability patterns
| Option | Use when | Avoid when |
|---|---|---|
| Self-consistency (N samples + voting) | Reasoning-heavy tasks, no labeled data for fine-tuning, confidence scores needed for routing. | Latency budget is tight (< 500ms) or task is simple enough that one call is already highly accurate. |
| Chain-of-thought (single sample) | Reasoning task where step-by-step output is the product itself, or cost is tightly constrained. | Single-sample accuracy is still insufficient after CoT; errors are correlated across reasoning paths. |
| RAG (retrieval-augmented generation) | Errors stem from missing or outdated factual knowledge, not from reasoning failures. | Problem is purely logical/mathematical -- retrieval doesn't help if all facts are already in the prompt. |
| Fine-tuning | You have 500+ labeled examples, task is narrow and high-volume, and single-call latency matters. | You lack labeled data, task changes frequently, or you need answers within days rather than weeks. |
| LLM-as-judge aggregation | Answers are free-form text where majority voting doesn't apply (e.g., summarization quality). | The judge model shares the same biases as the sampler, making the evaluation circular and unreliable. |
Code Example
# openai>=1.0.0
import os
from openai import OpenAI
from collections import Counter
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
def self_consistent_answer(question: str, n: int = 5) -> str:
answers = []
for _ in range(n):
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": question}],
temperature=0.8,
max_tokens=200,
)
answers.append(resp.choices[0].message.content.strip())
# Majority vote
winner, count = Counter(answers).most_common(1)[0]
print(f"Agreement: {count}/{n} — answer: {winner}")
return winner
print(self_consistent_answer("What is 17 * 24? Answer with only the number."))How this code works
This code demonstrates how to use "self-consistency" to get a more reliable answer from an AI model. Its main job is to ask a question multiple times, then aggregate these diverse responses to find the most common one, effectively using a "majority vote" to increase confidence in the result. This pattern is especially useful for factual questions where a single AI response might occasionally be incorrect or inconsistent.
The self_consistent_answer function takes a question and n (the number of times to ask) as input. Inside a for _ in range(n): loop, it repeatedly interacts with the OpenAI API using client.chat.completions.create. It specifies model="gpt-4o-mini" for the AI and sets temperature=0.8. This temperature setting is a subtle but important choice; instead of a very low temperature (which often produces identical answers), 0.8 encourages the AI to generate slightly varied responses each time, allowing the self-consistency check to be truly effective. Each AI response is then stored in an answers list.
After collecting all n answers, the code uses Counter(answers).most_common(1)[0] to perform the majority vote. Counter efficiently counts the occurrences of each unique answer, and most_common(1) retrieves the answer that appeared most frequently. Finally, it prints the level of Agreement before returning the winner, which is the AI's most consistent answer across its multiple attempts.
Production-grade example
Concurrent async sampling, per-call retries, token logging, confidence score, and human-review routing.
# openai>=1.0.0, tenacity>=8.0
import asyncio
import logging
import os
import time
from collections import Counter
from openai import AsyncOpenAI, RateLimitError, APITimeoutError
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger(__name__)
client = AsyncOpenAI(api_key=os.environ["OPENAI_API_KEY"])
@retry(
retry=retry_if_exception_type((RateLimitError, APITimeoutError)),
wait=wait_exponential(multiplier=1, min=2, max=30),
stop=stop_after_attempt(4),
)
async def _single_sample(prompt: str, model: str, temperature: float, timeout: float) -> str:
resp = await client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
temperature=temperature,
max_tokens=300,
timeout=timeout,
)
usage = resp.usage
log.info("sample tokens prompt=%d completion=%d", usage.prompt_tokens, usage.completion_tokens)
return resp.choices[0].message.content.strip()
async def self_consistent_classify(
prompt: str,
n: int = 7,
model: str = "gpt-4o-mini",
temperature: float = 0.8,
timeout: float = 15.0,
confidence_threshold: float = 0.6,
) -> dict:
start = time.monotonic()
tasks = [_single_sample(prompt, model, temperature, timeout) for _ in range(n)]
results = await asyncio.gather(*tasks, return_exceptions=True)
answers = [r for r in results if isinstance(r, str)]
errors = [r for r in results if isinstance(r, Exception)]
if errors:
log.warning("%d/%d samples failed: %s", len(errors), n, errors[0])
if not answers:
raise RuntimeError("All samples failed; cannot aggregate.")
counter = Counter(answers)
top_answer, top_count = counter.most_common(1)[0]
confidence = top_count / len(answers)
latency_ms = (time.monotonic() - start) * 1000
log.info(
"self_consistency answer=%r confidence=%.2f samples=%d/%d latency_ms=%.0f",
top_answer, confidence, len(answers), n, latency_ms,
)
return {
"answer": top_answer,
"confidence": confidence,
"needs_review": confidence < confidence_threshold,
"sample_count": len(answers),
"latency_ms": latency_ms,
}How this code works
This code implements the "self-consistency" pattern, designed to improve the reliability of AI model answers for tasks like classification or short responses. Its job is to overcome individual model inconsistencies or "hallucinations" by asking the AI the same question multiple times and then identifying the most frequently returned answer as the consensus.
The process begins with the _single_sample asynchronous function, which sends a single prompt to the AI model (client.chat.completions.create) and extracts the response. A key feature is the @retry decorator from tenacity, which automatically re-attempts the API call up to four times if it encounters common issues like RateLimitError or APITimeoutError. This robust error handling is crucial for real-world API interactions. The main self_consistent_classify function then generates multiple _single_sample tasks concurrently using asyncio.gather. After collecting all results, it filters out any errors and uses collections.Counter to tally the successful answers. It determines the top_answer and calculates its confidence score, indicating how strongly the AI agreed on that specific response, and flags it as needs_review if confidence falls below a confidence_threshold.
Practice & master
Try the exercise, check your understanding, then mark this lesson mastered to track your path to pro.
Exercise
Build a self-consistent sentiment classifier that labels movie reviews as Positive, Negative, or Neutral. Run 5 samples per review, aggregate with majority vote, and return both the label and a confidence score. Test it on at least 3 reviews where at least one is ambiguous. Print the full vote distribution for each.
# openai>=1.0.0
import os
from collections import Counter
from openai import OpenAI
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
REVIEWS = [
"The cinematography was stunning but the plot left me completely cold.",
"Absolutely loved every second. Best film I've seen this year!",
"It was fine, I guess. Nothing special, nothing terrible.",
]
def classify_sentiment(review: str, n: int = 5) -> dict:
answers = []
prompt = f"""Classify this movie review as exactly one of: Positive, Negative, Neutral.
Reply with only the label.
Review: {review}"""
for _ in range(n):
# TODO: Call the LLM with temperature=0.8
# TODO: Append the stripped response to answers
pass
# TODO: Count votes with Counter
# TODO: Compute confidence as top_count / len(answers)
# TODO: Return dict with 'label', 'confidence', 'votes'
pass
for review in REVIEWS:
result = classify_sentiment(review)
print(f"Review: {review[:60]}...")
print(f" Label: {result['label']} (confidence: {result['confidence']:.0%})")
print(f" Votes: {result['votes']}\n")Quick check
You run 9 samples and get votes [A:5, B:3, C:1]. Two samples threw API errors and weren't included. What is the correct confidence score?
Why does self-consistency perform poorly when temperature is set to 0?
You're building a numeric extraction pipeline (pulling dollar amounts from invoices). Which aggregation strategy is most appropriate for self-consistency here?