Phase 2: Prompt Engineering & LLM Patterns

JSON mode, function calling & structured output schemas

Intermediate ~14 min read
Prerequisites
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

You know how sometimes you're building a really cool LEGO city, and you ask a friend to help out? If you just say, "Hey, build me something!" they might give you a spaceship, a tree, or even just a pile of bricks. That's a bit like how an AI talks – it's super smart and can say amazing things, but often it just gives you free-form chat. But what if you need that "something" to be a specific part of your city, like a house, so you can connect other things to it? You need a way for the AI to give you exactly what you need, in a way your computer can understand and use immediately, not just read.

One way to help the AI is like telling your friend, "Okay, whatever you build, it needs to be a 'building' and fit on a standard LEGO baseplate." They'll give you a valid building on a baseplate – it won't be a car or a tree. But it could still be a tiny shack, a giant tower, or even a weird alien structure, as long as it fits the "building on a baseplate" rule. This is like something called JSON mode for AI. It guarantees the AI gives you information in a perfectly structured box (we call it JSON), but the exact details inside that box are still pretty flexible.

Now, what if you need a very specific piece for your city, like the 'fire station' with a big garage door, a helipad, and a sign? You'd give your friend a detailed instruction manual – a blueprint for exactly what a fire station needs to look like. This instruction manual is like what programmers call a structured output schema or function calling. You're giving the AI a precise plan for the information it should create. It doesn't just make any building; it follows your instructions to make a fire station. Some advanced AI tools can even make sure the AI builds it exactly by the manual, without any mistakes, right from the start!

Why is this super important? Well, with that perfectly built fire station, you can snap it right into your LEGO city. It connects to the roads, fits next to the police station, and is ready for the fire truck. If the AI gives you information about a customer's order for your online toy shop, and it’s perfectly structured like your instruction manual, your computer can instantly add it to your database, calculate the total price, and even print a shipping label, all without you having to sort through a messy pile of words. This means you can build bigger, more reliable AI tools that work every single time, without guessing.

The mental model: three layers of structure enforcement

Think of structured output as a spectrum with increasing strength of guarantee. At the bottom is plain prompting: you ask the model to "respond in JSON" and hope for the best. One tier up is JSON mode, which guarantees syntactically valid JSON but says nothing about which keys exist or what their types are. At the top are strict schemas, either via function calling with tool_choice: required or OpenAI's json_schema response format, which constrain the model's output token by token (or at decode time via constrained decoding, covered in a sibling lesson). For application integration you almost always want at least the middle tier, and usually the top.

Under the hood, function calling works by injecting the tool definitions into the model's context (consuming tokens, so schema verbosity has a real cost) and then fine-tuning or RLHF-training the model to produce well-formed JSON that satisfies those schemas. The model is not executing anything. When you set tool_choice: { type: 'function', function: { name: 'X' } }, you force the model to populate that function's argument schema and return nothing else. When you leave it as auto, the model decides whether a tool call is appropriate, which introduces a whole class of failure modes (more on that below).

A real scenario: document extraction pipeline

Imagine you're building an accounts-payable automation tool. Users upload PDFs; you extract text and pipe it through GPT-4o to pull out vendor name, line items, total amount, currency, and due date. You need to insert these rows into a Postgres table. With plain JSON mode, you still have to write defensive checks: is line_items an array? Does each item have a unit_price key? With a strict function-calling schema, the model either produces valid structured output or the API call errors, and you handle the error once at the integration layer rather than scattered through your parsing code. A senior engineer would map the JSON Schema directly to a Pydantic model so that model_validate(args) is the only parsing step needed. If validation fails, you retry with a corrected prompt or fall back gracefully.

Tradeoffs vs alternative approaches

You have four realistic options: (1) prompt-only JSON, (2) JSON mode, (3) function calling / tool use, (4) constrained decoding at the model-hosting layer. Prompt-only is fine for one-off scripts and prototyping. JSON mode is available on most providers and adds almost no latency, but you're still guessing at schema correctness. Function calling is the right default for production code against hosted APIs: it's well-supported on OpenAI, Anthropic (as tool use), and Google Gemini. Constrained decoding with libraries like Outlines or Guidance is powerful for self-hosted models where you control the inference stack, but adds infrastructure complexity. The sibling lesson on constrained decoding covers that path in depth.

One underappreciated tradeoff: schema verbosity versus token cost. A deeply nested JSON Schema with dozens of fields can add 400-800 tokens to every request, which matters when you're running 50k requests a day. Flatten schemas where possible, and consider splitting a giant schema into two sequential calls if complexity balloons.

What changes at scale

At 10 users you call the API synchronously and print results. At 10k requests per day you move to async batching (see aidev-python-async), add structured logging of every tool call and its parsed output, and monitor schema validation failure rates as a metric. A spike in Pydantic validation errors usually means the model started drifting from the schema, often after a provider-side model update. Pin your model version (gpt-4o-2024-11-20, not gpt-4o) in production to avoid surprise behavior changes.

At 10M requests per day the main concerns shift to cost per token (your schema is always in the context), latency percentiles, and fallback strategies. You'll want to cache identical or near-identical structured outputs, consider smaller models for simpler extraction tasks, and build a circuit breaker that falls back to a rule-based parser when the LLM error rate exceeds a threshold. Schema validation failures should go to a dead-letter queue for human review, not silently dropped.

Key Takeaways

  • Use JSON mode for valid syntax; use function calling or strict schemas to enforce structure and field types.
  • Define your schema in JSON Schema format; map it directly to a Pydantic model for type-safe Python access.
  • Function calling does not execute functions -- your code does; the model only produces the arguments.
  • Prefer strict schema enforcement over prompt-only instructions; prompts drift, schemas don't.

Pro tips

  • Pin your model version in production (gpt-4o-2024-11-20, not gpt-4o). Providers silently roll models, and a schema that worked perfectly on one checkpoint can start drifting on the next.
  • Your JSON Schema definition doubles as your Pydantic model spec. Write the Pydantic model first, then generate the JSON Schema from it with MyModel.model_json_schema() to keep them in sync automatically.
  • When tool_choice is auto, the model may decide no tool call is needed and return plain text instead. If your downstream code always expects a tool call, set tool_choice: required or force a specific function to eliminate that failure mode entirely.
  • Token cost for the tool definition scales with schema complexity and request volume. For high-throughput pipelines, benchmark the token overhead of your schema and consider splitting a complex extraction into two simpler calls if the schema pushes past 500 tokens.

Common pitfalls

  • Mistake: Using JSON mode and assuming the schema is enforced. Fix: JSON mode only guarantees valid JSON syntax; add explicit Pydantic validation after parsing to catch missing or wrong-typed fields.
  • Mistake: Leaving tool_choice as auto in extraction pipelines. Fix: Force the specific function with tool_choice: {type: 'function', function: {name: '...'}} so the model cannot return plain text instead of structured data.
  • Mistake: Defining the JSON Schema in one place and the Pydantic model separately, letting them drift. Fix: Generate the JSON Schema from the Pydantic model with model_json_schema() so there is one source of truth.
  • Mistake: Catching all exceptions with a bare except Exception. Fix: Catch specific types (ValidationError, RateLimitError, APITimeoutError) and handle each differently; silent swallowing hides real schema failures.

When to use JSON mode vs function calling vs strict schema

Option Use when Avoid when
Prompt-only JSON Prototyping or one-off scripts where parsing failure is acceptable. Any production path where downstream code depends on specific fields or types.
JSON mode You need valid JSON syntax and the schema is simple enough to validate cheaply yourself. You need guaranteed field presence and types; schema varies call-to-call.
Function calling / tool use Production extraction, classification, or action-selection where schema correctness is required. You need streaming token-by-token output and can't wait for a complete tool call object.
Strict json_schema response format (OpenAI) You want schema enforcement without the function-calling abstraction overhead; OpenAI API only. You are targeting multiple providers or need the model to choose among multiple tools.
Constrained decoding (Outlines, Guidance) Self-hosted models where you control inference and need hard guarantees with no retries. Using hosted APIs; adds significant infrastructure overhead for hosted setups.

Code Example

python
# openai>=1.30.0
import openai, json

client = openai.OpenAI()  # reads OPENAI_API_KEY from env

tools = [
    {
        "type": "function",
        "function": {
            "name": "extract_invoice",
            "description": "Extract invoice fields from text.",
            "parameters": {
                "type": "object",
                "properties": {
                    "vendor": {"type": "string"},
                    "amount_usd": {"type": "number"},
                    "due_date": {"type": "string", "format": "date"},
                },
                "required": ["vendor", "amount_usd", "due_date"],
            },
        },
    }
]

resp = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Invoice from Acme Corp, $1,200 due 2025-08-01."}],
    tools=tools,
    tool_choice={"type": "function", "function": {"name": "extract_invoice"}},
)

args = json.loads(resp.choices[0].message.tool_calls[0].function.arguments)
print(args)  # {'vendor': 'Acme Corp', 'amount_usd': 1200.0, 'due_date': '2025-08-01'}

How this code works

This code demonstrates how to use an AI model to extract specific, structured information from a piece of text. It sets up the openai client to interact with an AI model. A critical part is defining tools, which specifies a function named extract_invoice. This function acts like a blueprint, describing what data points (like vendor, amount_usd, and due_date) should be extracted and what type each should be (e.g., string, number). The parameters for this function are defined using a JSON Schema, which tells the AI exactly the structure it needs to output. The required field ensures these specific data points must always be provided.

Next, the code sends a user messages to the gpt-4o model, along with the defined tools. The tool_choice parameter is crucial here; it explicitly tells the model to always attempt to use the extract_invoice function. Without this, the AI might simply respond with text, not structured data. The AI then processes the message and responds with a tool_calls object containing the extracted data as a string within its arguments. Finally, json.loads converts this string into a usable Python dictionary, which is then printed, showcasing the successful extraction of structured invoice details directly from the plain text input.

Production-grade example

Adds pinned model version, timeout, typed retries, token logging, Pydantic validation, and graceful degradation.

python
# openai>=1.30.0, pydantic>=2.0, tenacity>=8.0
import json
import logging
import os
import time
from typing import Any

import openai
from pydantic import BaseModel, ValidationError
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type

logger = logging.getLogger(__name__)

client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"], timeout=15.0)

INVOICE_TOOL = {
    "type": "function",
    "function": {
        "name": "extract_invoice",
        "description": "Extract structured invoice fields from unstructured text.",
        "parameters": {
            "type": "object",
            "properties": {
                "vendor": {"type": "string"},
                "amount_usd": {"type": "number"},
                "due_date": {"type": "string", "format": "date"},
                "line_items": {
                    "type": "array",
                    "items": {
                        "type": "object",
                        "properties": {
                            "description": {"type": "string"},
                            "unit_price": {"type": "number"},
                            "quantity": {"type": "integer"},
                        },
                        "required": ["description", "unit_price", "quantity"],
                    },
                },
            },
            "required": ["vendor", "amount_usd", "due_date", "line_items"],
        },
    },
}

class LineItem(BaseModel):
    description: str
    unit_price: float
    quantity: int

class Invoice(BaseModel):
    vendor: str
    amount_usd: float
    due_date: str
    line_items: list[LineItem]

@retry(
    retry=retry_if_exception_type((openai.RateLimitError, openai.APITimeoutError)),
    wait=wait_exponential(multiplier=1, min=2, max=30),
    stop=stop_after_attempt(4),
)
def extract_invoice(text: str) -> Invoice | None:
    start = time.monotonic()
    try:
        resp = client.chat.completions.create(
            model="gpt-4o-2024-11-20",  # pinned version
            messages=[{"role": "user", "content": text}],
            tools=[INVOICE_TOOL],
            tool_choice={"type": "function", "function": {"name": "extract_invoice"}},
        )
        latency_ms = (time.monotonic() - start) * 1000
        usage = resp.usage
        logger.info(
            "llm_call",
            extra={
                "model": resp.model,
                "prompt_tokens": usage.prompt_tokens,
                "completion_tokens": usage.completion_tokens,
                "latency_ms": round(latency_ms, 1),
            },
        )
        raw_args = resp.choices[0].message.tool_calls[0].function.arguments
        return Invoice.model_validate(json.loads(raw_args))
    except openai.BadRequestError as exc:
        logger.error("bad_request", extra={"error": str(exc)})
        return None  # graceful degradation
    except ValidationError as exc:
        logger.error("schema_validation_failed", extra={"errors": exc.errors()})
        return None

How this code works

This code demonstrates how to reliably extract structured data, specifically invoice details, from unstructured text using an AI model and enforce a strict output schema. Its job in the lesson is to illustrate the combined power of OpenAI's function calling feature with Python's pydantic for robust data validation.

The code sets up an INVOICE_TOOL dictionary, which defines the expected structure for invoice data, including fields like vendor, amount_usd, due_date, and a list of line_items. This definition guides the AI model on what kind of structured JSON it should generate. Complementing this, LineItem and Invoice BaseModel classes from pydantic provide a Python-native way to represent and rigorously validate the AI's output. The extract_invoice function orchestrates the AI interaction. It calls client.chat.completions.create with the user's text and crucially specifies tools=[INVOICE_TOOL] and tool_choice to force the AI to use the predefined extract_invoice function. This tool_choice parameter is a subtle but vital aspect; it guarantees the AI must respond by generating structured arguments adhering to the INVOICE_TOOL schema. The AI's raw JSON response is then loaded and validated by Invoice.model_validate, ensuring the data is correct before being used, and tenacity automatically retries the AI call for transient errors.

Practice & master

Try the exercise, check your understanding, then mark this lesson mastered to track your path to pro.

Exercise

Build a function classify_support_ticket(text: str) -> TicketClassification that calls GPT-4o with a forced function call. The output schema must include category (one of: billing, technical, general), priority (1-3), and summary (string, max 20 words). Validate the result with Pydantic before returning it.

python
# openai>=1.30.0, pydantic>=2.0
import json
import os
import openai
from pydantic import BaseModel, Field

client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])

# TODO: Define a Pydantic model TicketClassification with
# category, priority, and summary fields and appropriate constraints.

# TODO: Define the CLASSIFY_TOOL dict with the matching JSON Schema.

def classify_support_ticket(text: str):
    # TODO: Call client.chat.completions.create with
    # model="gpt-4o", tools=[CLASSIFY_TOOL], and
    # tool_choice forcing the classify_ticket function.
    pass
    # TODO: Parse the tool call arguments and validate with
    # TicketClassification.model_validate(...).
    # Return the validated model instance.

if __name__ == "__main__":
    result = classify_support_ticket(
        "My invoice was charged twice this month and I need a refund urgently."
    )
    print(result)

Quick check

  1. You set response_format: { type: 'json_object' } (JSON mode). The model returns {"result": null}. What does this tell you?

  2. What actually happens when an LLM 'calls a function' during a function-calling API request?

  3. Your extraction pipeline uses tool_choice: 'auto' and intermittently receives plain text instead of a tool call. What is the simplest fix?

Self-check: Explain to a colleague the difference between JSON mode and function calling with a forced tool choice, describe one failure mode each introduces, and sketch how you would use Pydantic to bridge the function call output into a type-safe Python object.

Resources

Related Articles

Hand-picked reads to go deeper on structured output & parsing.

Back to AI Software Developer Roadmap