Chain-of-thought prompting works because of how autoregressive generation actually functions. A model generates one token at a time, and each token is conditioned on everything that came before it. When you force the model to emit reasoning tokens first, those tokens become part of the context window for the final answer token. Good intermediate reasoning steers the model's attention toward the relevant numbers, conditions, and logical branches. The model isn't actually "thinking" in a human sense; it's doing very fast pattern matching over its training distribution, but the patterns that produce correct multi-step answers are reliably associated with step-by-step exposition in the training data.
The practical difference between zero-shot and few-shot CoT is about steering quality, not mechanism. Zero-shot CoT adds a phrase like "Let's think step by step" or "Think through this carefully before answering." This works well on models trained with RLHF instruction-tuning (GPT-4, Claude, Gemini Pro) because those models have seen enormous amounts of step-by-step reasoning in training and in fine-tuning demonstrations. Few-shot CoT adds two to five complete examples with full reasoning traces written out. Use few-shot when your domain is specialized (tax law edge cases, a proprietary schema, unusual arithmetic conventions), when zero-shot CoT still makes category errors, or when you need the reasoning format to match a specific style for downstream parsing.
A real-world scenario: you are building an expense-approval agent that reads a corporate policy document and decides whether a submitted expense is compliant. The policy has nested conditions: international travel requires both a manager and CFO signature, unless the trip is under 48 hours, in which case manager-only suffices, unless the total exceeds $5,000. Without CoT, GPT-4o-mini at temperature 0 will occasionally collapse the conditions and approve or reject incorrectly. With CoT, the model writes out which conditions apply, checks them one by one, and almost always gets it right. More importantly, when it does fail, you have a trace to read. A senior engineer approaches this by (1) writing a zero-shot CoT prompt, (2) building a small test set of 20 labeled edge cases, (3) measuring accuracy, and (4) upgrading to few-shot CoT only if zero-shot misses more than two or three cases. This keeps prompts lean until complexity demands more.
CoT has real tradeoffs versus alternatives. Structured output prompting (JSON mode, function calling) is better when you need deterministic schema compliance and the task is not reasoning-heavy. Tool use (giving the model a calculator or a code interpreter) is better than CoT for arithmetic, because even with CoT a model can make a multiplication error in a long chain. For pure classification tasks (sentiment, intent detection) where the label space is small and examples are clear, few-shot prompting without explicit CoT is faster and cheaper. Where CoT wins is when the path to the answer is genuinely complex and the reasoning chain itself contains useful information for evaluation or audit.
Scale changes the calculus. At 10 users, extra tokens from a reasoning trace cost pennies and the debuggability is worth it. At 10,000 users making 50 requests each per day, a 400-token reasoning trace on a gpt-4o call might add $0.002 per request, which is $1,000 per day at that volume (rough illustration, verify current pricing). At that scale you start asking: can I use a smaller, cheaper model for the reasoning step and a fast model for the final answer? Can I cache CoT outputs for identical or near-identical inputs? Can I move reasoning-heavy steps to async pre-processing rather than the hot path? At 10 million users, you are likely distilling the CoT behavior into a fine-tuned model entirely, so the inference path no longer needs explicit prompting. Understanding this progression helps you architect prompts that stay maintainable as you grow.
Key Takeaways
- Instruct the model to reason step-by-step before answering to reduce multi-step errors.
- Use zero-shot CoT ('think step by step') for quick wins; few-shot CoT for higher stakes.
- Separate the reasoning trace from the final answer to make parsing and evaluation reliable.
- CoT increases token usage and latency; skip it for simple factual or classification tasks.
Pro tips
- Put the output format constraint at the END of the prompt, after the reasoning instruction. Models trained with RLHF satisfy the most recent instruction most reliably, so 'Verdict: APPROVED or REJECTED on the last line' at the bottom beats burying it in the middle.
- Use temperature=0 for CoT on tasks where correctness matters. Sampling variance compounds across a long reasoning chain: a slightly off step early produces a plausible but wrong path, and the model commits to it. Determinism is free accuracy on reasoning tasks.
- Parse the final answer by regex on a clearly delimited line, not by searching the full reasoning trace for any occurrence of your label words. Reasoning traces often contain phrases like 'this would be REJECTED if...' that break naive string matching.
- When evaluating CoT quality at scale, log the full reasoning trace alongside the verdict. A wrong verdict with a correct reasoning trace signals a parsing bug; a wrong verdict with a wrong trace signals a prompt or model issue. They require completely different fixes.
Common pitfalls
- Mistake: Asking for chain-of-thought reasoning on simple lookups or single-step classifications. Fix: Reserve CoT for tasks with 3+ logical steps. Extra tokens on simple tasks inflate cost and occasionally introduce confusion.
- Mistake: Not separating the reasoning trace from the final answer before passing it downstream. Fix: Require a structured delimiter like 'Answer:' or 'Verdict:' at the end and parse from there, never from the middle of the trace.
- Mistake: Using a high temperature with CoT on arithmetic or logic tasks. Fix: Set temperature=0. Reasoning chains are sensitive to early token variance; determinism is almost always better here.
- Mistake: Assuming CoT fixes arithmetic reliably. Fix: For precise calculations, give the model a code interpreter or calculator tool. CoT reduces arithmetic errors but does not eliminate them on long chains.
When to use CoT vs alternative prompting approaches
| Option | Use when | Avoid when |
|---|---|---|
| Zero-shot CoT | Multi-step reasoning task, no labeled examples, modern instruction-tuned model (GPT-4, Claude 3). | Simple single-step tasks, latency-critical paths, or specialized domains where the model lacks relevant training patterns. |
| Few-shot CoT | Domain-specific reasoning with unusual conventions; zero-shot CoT still makes category or format errors. | You have no high-quality labeled examples; bad examples in few-shot CoT actively hurt performance. |
| Tool use (code interpreter / calculator) | Precise arithmetic or data transformation is required; correctness is non-negotiable. | Your hosting environment disallows tool calls or the reasoning is qualitative/logical rather than numeric. |
| Direct answer prompting (no CoT) | Classification with a small, clear label space; single-step factual retrieval; extremely latency-sensitive path. | The answer depends on evaluating multiple interacting conditions or performing sequential calculations. |
Code Example
# openai >= 1.0.0
from openai import OpenAI
client = OpenAI() # reads OPENAI_API_KEY from env
prompt = """
A server processes 240 requests per minute at peak load.
Each request takes on average 150ms of CPU time.
The server has 4 CPU cores. Is the server CPU-bound at peak load?
Think step by step, then give a final yes/no answer on the last line
formatted exactly as: Answer: yes or Answer: no
"""
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
temperature=0,
)
text = response.choices[0].message.content
print(text)
# Parse out just the final answer
for line in text.splitlines():
if line.strip().lower().startswith("answer:"):
print("Extracted answer:", line.split(":", 1)[1].strip())
breakHow this code works
This code demonstrates "chain-of-thought" prompting, a technique to guide an AI model through complex reasoning by asking it to "Think step by step" before giving a final answer. Specifically, it presents the AI with a word problem about server capacity and then extracts just the ultimate "yes" or "no" conclusion from the model's full response.
The script begins by importing the OpenAI library and setting up a client to interact with the AI service; this client automatically reads the OPENAI_API_KEY from the environment. The prompt variable holds the core instruction: the word problem along with the crucial direction for step-by-step thinking and the required final answer format. A subtle but important detail is the temperature=0 setting used when calling client.chat.completions.create. This choice makes the AI's reasoning process more deterministic and less creative, which is often preferred for logical problem-solving tasks like this.
After the AI generates its response, text = response.choices[0].message.content extracts the complete output, which includes both the detailed thought process and the final answer. The subsequent for line in text.splitlines(): loop then parses this multi-line response. It efficiently searches for any line starting with "Answer:" (regardless of capitalization or surrounding spaces) to isolate and print only the extracted "yes" or "no," showing how to programmatically pull specific structured data from a free-form AI output.
Production-grade example
Adds retries, timeouts, structured logging, regex answer extraction, and graceful degradation on missing verdict.
# openai >= 1.0.0, tenacity >= 8.2
import os
import time
import logging
import re
from openai import OpenAI, RateLimitError, APITimeoutError, APIStatusError
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger(__name__)
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"], timeout=30.0)
COT_SYSTEM = (
"You are a compliance assistant. Reason step by step through each condition, "
"then end your response with exactly one line: Verdict: APPROVED or Verdict: REJECTED"
)
@retry(
retry=retry_if_exception_type((RateLimitError, APITimeoutError)),
wait=wait_exponential(multiplier=1, min=2, max=60),
stop=stop_after_attempt(4),
)
def evaluate_expense(policy: str, submission: str) -> dict:
user_msg = f"Policy:\n{policy}\n\nExpense submission:\n{submission}\n\nAnalyze compliance."
t0 = time.monotonic()
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": COT_SYSTEM},
{"role": "user", "content": user_msg},
],
temperature=0,
max_tokens=600,
)
latency_ms = round((time.monotonic() - t0) * 1000)
usage = response.usage
log.info(
"cot_expense_eval prompt_tokens=%d completion_tokens=%d latency_ms=%d",
usage.prompt_tokens, usage.completion_tokens, latency_ms,
)
full_text = response.choices[0].message.content
verdict_match = re.search(r"Verdict:\s*(APPROVED|REJECTED)", full_text, re.IGNORECASE)
if not verdict_match:
log.warning("No verdict line found; defaulting to REJECTED. Raw: %s", full_text[:200])
verdict = "REJECTED"
else:
verdict = verdict_match.group(1).upper()
return {
"verdict": verdict,
"reasoning": full_text,
"prompt_tokens": usage.prompt_tokens,
"completion_tokens": usage.completion_tokens,
"latency_ms": latency_ms,
}How this code works
This code demonstrates a chain-of-thought (COT) prompting strategy to act as a compliance assistant, evaluating expense submissions against a policy. Its job in the lesson is to show how an AI can be instructed to reason step-by-step and then provide a structured answer. It uses the openai library to send policy and submission details to an AI model, specifically gpt-4o-mini, requesting a structured reasoning process before a clear verdict. The setup involves logging for monitoring and initializing the OpenAI client with an OPENAI_API_KEY. The core instruction for the AI, defined in COT_SYSTEM, guides it to think through each condition and conclude with a specific Verdict: line.
The evaluate_expense function orchestrates this interaction. It constructs a user message, then calls client.chat.completions.create, passing the system prompt and user input. Setting temperature=0 ensures consistent, less creative AI reasoning. For robust API calls, the function uses a @retry decorator from tenacity, which automatically re-attempts the request if RateLimitError or APITimeoutError occur. After receiving the AI's response, the code extracts the full reasoning. A subtle but important detail is the use of re.search to find the final Verdict:. If the AI, despite instructions, fails to provide this line, the code intelligently defaults the outcome to "REJECTED" and logs a warning, ensuring a decision is always returned. Finally, it bundles the verdict, full reasoning, and token usage details into a dictionary.
Practice & master
Try the exercise, check your understanding, then mark this lesson mastered to track your path to pro.
Exercise
Build a function that takes a short word problem as input and returns a dict with 'reasoning' (the model's step-by-step trace) and 'answer' (the extracted final answer). Test it on at least two problems: one arithmetic and one conditional logic problem. Compare accuracy with and without a CoT instruction by calling the function both ways.
# openai >= 1.0.0
from openai import OpenAI
import os
import re
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
def solve_problem(problem: str, use_cot: bool = True) -> dict:
# TODO: Build the prompt.
# If use_cot is True, instruct the model to think step by step
# and end with a line formatted as "Answer: <value>"
# If use_cot is False, just ask for the answer directly.
prompt = ""
# TODO: Call the API with temperature=0
response = None
# TODO: Extract 'reasoning' (full text) and 'answer' (parsed from 'Answer:' line)
# If no Answer: line is found, set answer to "UNKNOWN"
return {
"reasoning": "",
"answer": "",
}
# Test problems
problems = [
"A store buys apples for $0.30 each and sells them for $0.45 each. If they sell 200 apples, what is the profit?",
"If Alice is older than Bob, and Bob is older than Carol, is Alice definitely older than Carol?",
]
for p in problems:
print("--- CoT ON ---")
print(solve_problem(p, use_cot=True))
print("--- CoT OFF ---")
print(solve_problem(p, use_cot=False))Quick check
Why does adding 'think step by step' improve model accuracy on multi-step problems?
You have a task where users ask the model to add up five numbers. Should you use CoT?
A CoT prompt produces correct reasoning but you parse the wrong final answer. What is the most likely cause?