Robust parsers for edge cases & malformed responses

The mental model here is a parsing pipeline, not a single function call. Think of raw LLM output as untrusted user input. You would never take a string from a web form and insert it directly into a SQL query. Same logic applies: you take raw text, put it through a series of transformations that each handle a specific class of failure, and only trust the output once it has passed all gates.

The failure modes you need to handle fall into three categories. First, framing failures: the model wraps the JSON in a markdown code fence (```json ...``), adds a preamble like "Here is your JSON:", or appends a trailing explanation after the closing brace. These are fixable with a regex or string slicing pass before you even attempt to parse. Second, syntax failures: trailing commas, unquoted keys, single-quoted strings, missing closing brackets, or control characters inside string values. The json-repair` library (pip installable, MIT licensed) handles most of these by walking the token stream and guessing intent. Third, schema failures: the JSON is syntactically valid but the model put a string where you expected an integer, omitted a required field, or nested objects one level deeper than your schema expects. These require schema validation, and Pydantic v2 is the practical standard for Python.

A real-world scenario: you are building a product catalog ingestion pipeline. You send batches of raw product descriptions to GPT-4o and ask it to return structured records with fields like sku, price_usd, category, and in_stock. Works great in dev with clean descriptions. In production, a vendor sends you a description that is 3800 tokens long, and the model truncates mid-response because you forgot to budget output tokens correctly. Now you have a half-formed JSON object. A naive parser crashes. A robust parser detects the truncation (no closing brace), calls repair_json, gets a partial but usable record, validates that the required fields that did make it through are correct, logs the truncation event with the original prompt and model response, and either queues a retry with a shorter input or accepts the partial record with a flag. The pipeline continues. The on-call engineer sees a structured alert in Datadog, not a 500 error storm.

The tradeoff versus constrained decoding (covered in aidev-structured-output-decoding) is worth naming explicitly. Constrained decoding, as implemented by tools like outlines or guidance, forces the model to only emit tokens that keep the output valid according to a grammar. It eliminates syntax failures entirely. But it requires running the model yourself or using a provider that exposes logit-bias-level control. OpenAI's structured outputs feature is the hosted equivalent and handles most cases. The gap where post-hoc repair still wins: you are calling a third-party API that does not support constrained decoding, you are processing legacy responses stored in a database, or you need to handle model output from multiple providers in a single pipeline. Defense in depth means even if you use structured output mode, you still validate with Pydantic, because the provider's guarantee is about syntax, not your business schema.

What changes at scale matters a lot. At 10 users, a try/except that logs and returns None is fine. At 10k users, you need to distinguish error categories so you can route retriable errors (truncation, temporary model degradation) to a retry queue and permanent errors (model consistently misunderstood the schema) to a dead-letter queue for human review. You want metrics: parse success rate, repair success rate, validation failure rate, broken down by model and prompt version. At 10M users, you are likely running this in a worker pool, parsing is on the hot path, and you need to ensure json-repair is not the bottleneck. Profile it: the library does string manipulation and is fast enough for most workloads, but if you are parsing 50k responses per second, consider a Rust-based JSON parser with repair for the hot path and fall back to Python only on failure. You also want circuit breakers: if the parse failure rate for a given model or prompt version spikes above a threshold, stop sending requests and alert, because something upstream changed.

Cost and latency implications: retrying a failed parse means you are paying for another API call and adding latency. Budget your retry budget carefully. One retry is usually worth it. Three retries on every failure will 3x your tail latency and cost. A smarter approach is to classify the failure first: if repair produced a valid schema-conforming result, accept it without a retry. Only retry if you genuinely lost required fields and the failure looks like truncation rather than a schema misunderstanding. Log the repair diff so you can audit whether the repaired output was actually correct.