AI in Production

I spent six months watching an AI feature I helped build slowly bleed money in production. The demo had been flawless. The VP loved it. And then real users showed up, and everything we’d assumed about cost, latency, and reliability turned out to be wrong.

This is the operational playbook I wish existed when I first pushed an LLM-powered feature to real users. I’m breaking down the architecture decisions, cost dynamics, latency traps, observability patterns, and team structures that separate production AI systems from impressive demos. If you’re shipping AI to real users in 2026, these are the questions that actually matter.

Why Most AI Projects Never Make It to Production

Here’s what nobody wants to admit out loud: the vast majority of AI projects stall before reaching real users. A 2024 RAND Corporation study found that roughly 80% of AI projects fail. That’s significantly worse than the already-painful software industry average. I’ve watched this play out across multiple organizations, and the reasons are almost never about model quality.

They’re about everything around the model.

The gap between a working notebook and a production AI system is enormous. In a demo, you control the inputs. In production, users type gibberish, send 40-paragraph prompts, and hit your API at 3 AM on a Saturday when your on-call engineer is asleep. The model that scored 92% on your eval suite suddenly hallucinates customer data or takes 8 seconds to respond because you never load-tested the retrieval-augmented generation pipeline under concurrent load.

Anthropic’s engineering team puts it well in their guide to building effective agents: “We recommend finding the simplest solution possible, and only increasing complexity when needed.” Sounds obvious. In practice, I’ve seen teams reach for agent orchestration frameworks before they’ve even validated that a single well-prompted LLM call solves their problem. The result is AI tech debt that compounds faster than traditional software debt because you’re debugging probabilistic systems with deterministic intuitions.

The teams that succeed treat production readiness as a first-class design constraint from day one. Not a phase you bolt on after the demo impresses the VP.

The Real Cost of Running LLMs at Scale

Cost is the silent killer of AI in production. I watched a team build a beautiful RAG pipeline, ship it to beta users, then get a $47,000 cloud bill for the first month. Nobody had done the math. The economics change completely when you move from developer testing (tens of queries per day) to production traffic (thousands or millions).

Here’s what the cost landscape actually looks like in 2026:

The real unlock is understanding that not every request needs your most powerful model. I wrote about this when analyzing how Netflix’s Headroom system cuts AI agent costs by 10x. Their approach routes simple classification tasks to lightweight models and reserves expensive reasoning calls for genuinely complex queries. This kind of intelligent routing is the difference between a sustainable production system and one that bleeds money until someone pulls the plug.

LLM cost optimization isn’t glamorous work. It’s the work that determines whether your AI feature survives past the pilot phase. If you’re evaluating serving infrastructure, the vLLM vs Ollama comparison breaks down when to use production-grade serving versus developer-friendly tooling.

Latency: The Metric That Trumps Everything

Here’s the thing nobody’s saying about AI in production: your users don’t care about your model’s benchmark scores. They care about how fast it responds.

I’ve written about why AI latency matters more than intelligence, and the data is stark. In one analysis, a 232ms improvement in response time correlated directly with user engagement metrics. Google’s own research showed that a 500ms delay in search results causes a 20% drop in traffic. LLM responses that take 3-5 seconds feel broken to users conditioned by instant search.

Latency in production AI systems stacks up from multiple sources:

1. Model inference time — the raw compute to generate tokens, heavily dependent on model size and hardware
2. Retrieval pipeline overhead — if you’re using RAG, your vector database query time adds directly to perceived latency
3. Network hops — API-based models add round-trip time; self-hosted models trade this for infrastructure complexity
4. Prompt construction — assembling context, system prompts, and few-shot examples takes non-trivial time at scale
5. Output parsing and validation — structured output extraction, guardrail checks, response formatting
6. Orchestration overhead — multi-agent AI systems multiply latency by the number of sequential agent calls
7. Cold start penalties — serverless GPU instances or auto-scaled containers have initialization costs that destroy P99 latency

The LLM API latency benchmarks I’ve published show massive variance between providers, and even between different times of day on the same provider. When evaluating models for production, raw speed comparisons like Claude Haiku 4.5 vs GPT-4o Mini matter more than most teams realize. Your fastest model at acceptable quality often beats your smartest model at unacceptable latency.

Andrej Karpathy, former Tesla AI director and OpenAI founding member, has repeatedly emphasized that the future of AI is about making models smaller and faster, not just bigger and smarter. His work on efficient inference architectures reflects a production-first mindset that the industry is slowly waking up to.

Choosing Your Model Architecture for Production

The model selection conversation has grown up a lot. In 2023, it was “use GPT-4 for everything.” In 2026, the options are genuinely competitive and the decision tree actually matters.

For production workloads, you’re choosing between three architectural approaches:

Dense models like Llama 3 70B activate every parameter for every token. Predictable, well-understood, excellent tooling support. The tradeoff: you pay full compute cost regardless of query complexity. A simple classification task costs the same as a complex reasoning chain.

Mixture-of-Experts (MoE) models like Mixtral 8x22B activate only a subset of parameters per token. You get a larger total parameter count (and the knowledge that comes with it) at a fraction of the inference cost. I’ve seen MoE models deliver 90% of dense model quality at 40% of the compute cost for routing and classification tasks.

Distilled and quantized models sacrifice some capability for dramatic speed improvements. For many production use cases — entity extraction, sentiment classification, simple summarization — a well-fine-tuned 7B model outperforms a prompted 70B model because it’s been optimized for your specific task.

The decision framework I actually use: start with the cheapest model that might work, build rigorous evals, and only scale up model size when your evals prove you need it. This is the opposite of what most teams do. Most teams start with GPT-4 class models, build their prompts around that capability level, then discover they can’t afford it at scale. I’ve shipped enough features to know that starting cheap and scaling up is dramatically less painful than starting expensive and trying to cut costs later.

Harrison Chase, CEO of LangChain, has been vocal about this pattern: teams over-invest in model capability and under-invest in evaluation infrastructure. The teams that ship reliably to production have evals running in CI/CD before they have a UI.

The Observability Gap in Production AI

Traditional application monitoring tells you if your server is up and your response times are acceptable. Production AI needs an entirely different observability layer because the failure modes are fundamentally different.

A REST API either returns the right data or throws an error. An LLM can return confidently wrong answers with a 200 status code. This is the observability gap. It’s where I’ve seen the most production incidents in AI systems, and it’s where teams are least prepared.

What production AI observability actually requires:

• Trace-level logging of every LLM interaction — input prompt, output completion, token counts, latency, model version. You need this for debugging, but also for building your eval datasets over time.
• Semantic drift detection — monitoring whether your model’s outputs are changing character over time, especially after provider-side model updates you didn’t request. This one bites people. OpenAI swaps a model version, your carefully tuned prompts start producing different outputs, and you don’t notice for a week.
• Retrieval quality metrics — if you’re running RAG, tracking retrieval precision and recall is non-negotiable. A vector database returning irrelevant chunks is the root cause of most hallucination incidents I’ve investigated.
• Cost attribution per feature — knowing your total AI spend is useless. You need cost broken down by feature, by user segment, by prompt template. Without this, you’re flying blind on unit economics.
• Guardrail hit rates — if your safety filters are triggering on 15% of requests, that’s either a sign your filters are too aggressive or your users are adversarial. Either way, you need to know.

Tools like LangSmith, Arize Phoenix, and Weights & Biases have matured a lot for this purpose. But I’ve found that the best observability setups are partially custom-built. Every production AI system has domain-specific failure modes that off-the-shelf tools don’t catch.

The AI agent failure patterns I’ve documented show that most production incidents aren’t model failures. They’re integration failures, context window overflows, and cascading retry storms. Your observability needs to catch these at the system level, not just the model level.

Security and Safety in Production AI Systems

Shipping AI to production means accepting a new threat surface that most engineering teams aren’t trained for. Prompt injection gets all the attention, but it’s only the beginning.

Production AI security concerns fall into three categories:

Input attacks — prompt injection, jailbreaking, and adversarial inputs designed to make your model behave in unintended ways. Simon Willison, one of the most thoughtful voices on LLM security, has documented extensively how prompt injection remains fundamentally unsolved. Every mitigation is a heuristic, not a guarantee. If someone tells you they’ve “solved” prompt injection, they’re selling you something.

Data exfiltration — your RAG pipeline has access to internal documents, customer data, or proprietary information. A well-crafted prompt can sometimes extract information the user shouldn’t have access to. This isn’t theoretical. I’ve seen it in production security audits, and it’s terrifying how easy it can be.

Output safety — your model might generate harmful content, biased recommendations, or legally problematic statements. In regulated industries like healthcare or finance, a single bad output can trigger compliance violations.

The practical defense is layered: input validation, output filtering, permission-scoped retrieval, and comprehensive logging. None of these layers is perfect on its own. Together, they reduce risk to manageable levels.

If you’re building with AI agents that have tool-use capabilities, the attack surface multiplies. An agent that can execute database queries or call external APIs needs the same security rigor as any other service with those permissions. I’ve seen teams give agents admin-level database access in production because “it worked in testing.” That’s not engineering, that’s a ticking time bomb. The PocketOS database disaster is a cautionary tale of what happens when agent permissions aren’t properly scoped.

Building the Right Team for Production AI

The organizational failure mode I see most often: companies hire ML engineers to build models, then expect their existing platform team to figure out production deployment. This gap is where projects go to die.

Production AI requires a blend of skills that doesn’t map cleanly to traditional roles:

• ML engineers who understand model selection, fine-tuning, and evaluation
• Platform engineers who can build serving infrastructure, manage GPU clusters, and handle autoscaling
• Data engineers who build and maintain the retrieval pipelines, vector embeddings, and knowledge bases that feed your models
• Product engineers who understand what it means to ship probabilistic systems to users — loading states, confidence indicators, fallback behaviors when the model fails
• Security engineers who understand the novel threat models of AI systems

The most effective structure I’ve seen is a small, cross-functional AI platform team (3-5 people) that owns the shared infrastructure — model serving, observability, guardrails, cost management — while product teams own the application-level AI features built on that platform. This separation of concerns prevents every product team from reinventing the wheel on inference infrastructure.

Chip Huyen, author of Designing Machine Learning Systems, argues that the distinction between ML engineering and ML operations is dissolving. I think she’s right. The engineers who thrive in production AI are full-stack in a new sense: they understand the model, the serving infrastructure, and the product implications. The generative AI courses available today are getting better at teaching model development, but most still have significant gaps around production operations. If you’re hiring, look for engineers who’ve actually broken things in production, not just people who can fine-tune a model in a notebook.

The Infrastructure Stack That Actually Works

After working with multiple production AI deployments, the infrastructure stack I’ve seen succeed most consistently follows a clear pattern:

Serving layer: vLLM for self-hosted open models with PagedAttention for efficient memory management. For API-based models, a routing layer that can failover between providers and route by cost/latency/quality tradeoffs. Don’t hardcode a single provider. You’ll regret it when they have an outage at your worst possible moment.

Retrieval layer: A dedicated vector database like Qdrant or Milvus for semantic search, paired with traditional search for keyword matching. The Qdrant vs Chroma comparison is relevant if you’re choosing between embedded and standalone vector stores. Hybrid retrieval (vector + keyword) consistently outperforms either approach alone in my experience.

Orchestration layer: Keep it simple. Seriously. Anthropic’s guidance on building effective agents is worth repeating: use simple, composable patterns rather than complex frameworks. I’ve seen more production incidents caused by over-engineered agent frameworks than by under-engineered ones. The boring answer is the right one here.

Evaluation layer: Automated evals running on every prompt template change, every model version update, every retrieval pipeline modification. This is your CI/CD for AI. Without it, you’re deploying blind and hoping for the best.

Observability layer: Structured logging of all LLM interactions, cost tracking, latency percentiles (especially P99 — check the memory allocator impact on P99 latency if you’re running self-hosted inference), and semantic quality monitoring.

The local agentic coding workflows emerging in 2026 are interesting for development, but production systems need the rigor of managed infrastructure. The gap between what works on a developer’s machine and what survives production traffic is where most AI projects fail.

What Comes Next: Production AI in Late 2026 and Beyond

I think we’re at an inflection point. AI in production is moving from “can we make it work?” to “can we make it work reliably, affordably, and safely at scale?” Same maturation curve every technology goes through, just compressed into months instead of years.

Three trends I’m watching closely:

Model costs are in freefall. The cost per million tokens for frontier models has dropped by roughly 90% in the last 18 months. This isn’t a small shift. Features that were too expensive to ship six months ago are now viable. Use cases that only made economic sense for FAANG companies are opening up to startups. The economic calculus is being rewritten in real time.

Structured outputs are becoming standard. OpenAI, Anthropic, and Google all now support structured output modes that dramatically improve reliability for production use cases. When your model returns validated JSON instead of freeform text, your integration code gets simpler and your error rates drop. This is one of those unglamorous improvements that makes a massive practical difference.

The evaluation problem is the real bottleneck. We have good models. We have good serving infrastructure. What we don’t have is a scalable way to know if our AI features are actually working well for users. The teams investing in evaluation infrastructure today are going to have a massive advantage in 12 months. Everyone else will still be guessing.

The bottleneck for AI in production has shifted from model capability to engineering discipline. The models are good enough. The question is whether your team can build the systems around them.

If you’re building production AI systems right now, the boring advice is the right advice: invest in evals, monitor everything, start with the simplest architecture that works, and treat cost as a first-class engineering constraint. The teams that do this unglamorous work are the ones whose features are still running six months later. Everyone else will have a great demo and a Jira ticket labeled “decommission.”

Frequently Asked Questions About AI in Production

What percentage of AI projects actually make it to production?

Industry estimates vary, but the commonly cited figure is around 80% of AI projects fail to reach production. Based on what I’ve seen firsthand, the failure rate for LLM-based projects is potentially lower than traditional ML projects because the barrier to a working prototype is much lower. But the attrition from prototype to sustainable production system remains high, primarily due to cost, latency, and reliability issues rather than model capability.

How much does it cost to run an LLM in production?

It depends enormously on traffic volume, model choice, and architecture. A small application making 10,000 API calls per day to GPT-4o might spend $500-$1,500/month. A high-traffic application can easily hit $50,000+/month. Self-hosting open models on dedicated GPUs shifts costs to infrastructure but can reduce per-token costs by 10-50x at sufficient scale. The key is model routing — using cheaper models for simple tasks and reserving expensive models for complex ones.

Should I use an API-based model or self-host for production?

For most teams starting out, API-based models are the right call. They let you validate your use case without infrastructure investment. Consider self-hosting when: you need data sovereignty, your token volume makes APIs prohibitively expensive, you need latency guarantees the API can’t provide, or you need to run a fine-tuned model specific to your domain. Many mature production systems end up with a hybrid approach, and that’s fine.

What’s the biggest mistake teams make when deploying AI to production?

Skipping evaluation infrastructure. I’ve watched teams spend months building sophisticated RAG pipelines and agent workflows, then deploy with no systematic way to measure output quality. When things break — and they will — these teams can’t tell the difference between a model regression, a retrieval failure, and a prompt issue. Build your evals before you build your UI. I’m not being dramatic. This is the single highest-leverage thing you can do.

How do I handle hallucinations in production?

You can’t eliminate hallucinations. Full stop. But you can manage them. Use retrieval-augmented generation to ground responses in source documents. Implement citation mechanisms so users can verify claims. Add confidence thresholds — if the model’s retrieval scores are below a threshold, return a graceful fallback instead of a potentially hallucinated response. Log everything so you can identify hallucination patterns and adjust your retrieval pipeline or prompts accordingly.

Is RAG or fine-tuning better for production use cases?

RAG and fine-tuning solve different problems, and asking which is “better” is the wrong question. RAG is better when your knowledge base changes frequently, when you need citations, or when you need to scope access to specific documents per user. Fine-tuning is better when you need to change the model’s behavior, tone, or output format consistently, or when you need faster inference by baking knowledge into the model weights. Most production systems I’ve worked on benefit from both: a fine-tuned model that’s better at your specific task, augmented with RAG for current information.

How do I monitor AI quality in production?

Traditional uptime monitoring isn’t enough. You need semantic quality monitoring: automated evaluations that check output relevance, groundedness (for RAG systems), and safety on a sampling basis. Track user feedback signals like thumbs-up/down ratios, regeneration rates, and session abandonment. Set up alerts for drift in these metrics. Tools like LangSmith, Arize, and custom evaluation pipelines built on your own test cases are all valid approaches. The important thing is to pick one and actually use it.

What infrastructure do I need to run AI in production?

At minimum: a serving layer for model inference, a retrieval layer if using RAG, an evaluation framework, an observability stack, and a cost tracking system. For API-based setups, this can be relatively lightweight. For self-hosted models, add GPU management, model versioning, and autoscaling to the list. Start simple and add complexity only when your metrics tell you to. If you’re building infrastructure you don’t need yet, you’re wasting time you could spend on evals.