How to Do Non Deterministic AI System Testing [2026]

If you’re shipping LLM features, you’re already doing non deterministic AI system testing. You’re just doing it badly.

I don’t mean “you don’t have a dashboard.” I mean you’re probably using vibes as a release gate. One good-looking trace. A couple cherry-picked screenshots. Maybe a prompt file named final_v7_really_final.md.

The outcome you actually want is boring: a CI gate that catches real regressions without flaking every other merge. The prerequisite that trips teams up is also boring: you have to treat evaluation artifacts (cases, labels, rubrics, retrieval snapshots, prompts, tool versions) as versioned build inputs, not as “some prompts in a folder.”

I learned this building and running this site’s 7‑agent publishing pipeline. Deterministic gates before LLM review catch more than doubling the review model’s size. Same lesson here. If your gate is vibes, you will ship regressions.

What is non deterministic AI system testing (and why classic unit tests fail for LLMs/agents)?

Non deterministic AI system testing is the practice of verifying AI features whose outputs can legitimately vary run-to-run (sampling, backend churn, retrieval drift, tool timing) while still enforcing stable, product-critical behaviors.

Classic unit tests assume you can assert exact outputs: foo(3) == 7. LLM apps rarely behave like that. Even at temperature 0, you still get variance from:

• Vendor churn: silent model upgrades, new “thinking” modes, routing changes.
• Prompt edits: the “tiny copy tweak” that changes tool call order.
• Retrieval drift: the index refreshed, embeddings changed, the doc moved.
• Tool nondeterminism: time-based APIs, rate limits, retries, partial failures.

You’re testing a stochastic system. Your job is to build evidence that behavior stayed inside acceptable bounds.

If you’re building AI agents or any kind of production AI, the right mental model is: unit tests still matter, but release gates need a different toolbox.

Start with named cases, not random traffic

Random traffic feels “realistic.” It’s also a trap.

It changes every run, so you can’t compare baseline vs candidate cleanly. It hides selection bias because you end up showing the one trace that flatters your change. And it trains teams to argue about anecdotes instead of looking at system behavior.

Raju Dandigam says it bluntly: one passing agent run is not a release signal. He’s right. A single trace is evidence for one execution, not for the system across representative cases. Read it: Raju Dandigam.

How I define “named cases”

A named case is a stable test input with:

• an ID (refund-unknown)
• a scenario intent (“unknown order must not refund”)
• a reproducible input bundle (prompt, tool inputs, retrieval snapshot ID)
• explicit checks (hard and soft)

A practical starter set is 30–80 cases. Below ~30, you’re mostly measuring noise. Above ~200, teams stop maintaining it and it turns into a graveyard of half-relevant scenarios.

Make the suite explicit and reviewable. When it changes, you want a diff. When it fails, you want to reproduce that exact case.

This is also where prompt injection belongs. Security cases should be named, not “we ran a red-team prompt once and it seemed fine.”

The set is part of the test (golden sets that survive churn)

Your golden set is not “a JSONL file.” It’s a product artifact.

Ken W Alger makes a point I agree with: as AI codegen gets cheap, verification/specification becomes the durable artifact. That applies to LLM products too. The durable thing isn’t your prompt. It’s the executable spec that defines what “good” means. His post is worth your time: Ken W Alger.

Here’s how I build a golden set that survives weekly churn.

1) Split the golden set into tiers

I use three tiers because they map cleanly to CI lanes:

1. Smoke (10–20 cases): fast, cheap, fail-closed.
2. Gate (50–100 cases): your real release signal.
3. Nightly (200–1000 cases): drift detection and long-tail coverage.

If your model call costs $0.002 per run and your gate suite is 80 cases, a single run is ~$0.16. If you do 10 runs for tolerance bands (we’ll get there), that’s $1.60 per PR.

That’s not the expensive part.

The expensive part is shipping bad behavior and then doing the “how did this get out?” dance in Slack.

If you need to control LLM cost, do it with lane design and caching, not by pretending a single run is “good enough.”

2) Version everything that can churn

Every eval result should record:

• model_provider and model_id
• prompt_version (hash the system + developer prompt)
• temperature, top_p, max_tokens
• tool versions (including the tool schema)
• retrieval snapshot ID (or index commit)
• judge model ID + rubric version (if using LLM-as-judge)

If you can’t answer “what changed?” you can’t debug regressions.

This is one of those things where the boring answer is actually the right one. Treat it like build metadata.

3) Maintain the set like you maintain production code

Golden sets rot for three reasons:

• Labels get stale.
• The product spec changes but the tests don’t.
• People add cases only when something breaks.

My rule: every week, add 5–10 counterexample cases from production failures and near-misses. Then retire or rewrite 1–2 ambiguous cases that caused reviewer arguments.

Also, if you have RAG, do not hand-wave retrieval quality. Keep a dedicated retrieval eval layer and link it to answer quality. I go deep on that in RAG and retrieval-augmented generation.

Compare behavior as a cohort (baseline vs candidate, not “did it pass?”)

A single score hides the truth. Cohorts show it.

Cohort comparison means: run the same named cases on baseline and candidate, then analyze deltas:

• Which cases flipped from pass→fail?
• Which cases improved?
• Did failures cluster by intent (e.g., “refund approval required”)?
• Did tool behavior change (more retries, different tool order)?

This is where Raju’s “cohort deltas” framing is useful. You’re not chasing a green check. You’re producing evidence a human can review.

What counts as “real” vs “noise”

For deterministic checks (schema-valid JSON, tool must be called, forbidden tool never called), flips are usually real.

For semantic checks (summary quality, helpfulness, tone), you need tolerance bands. Otherwise you’ll spend your life arguing about one example.

Also, make deltas visible. I’ve watched teams hide eval diffs behind a dashboard nobody opens. Put it in the PR where the decision is being made.

If you’re already building observability, connect this to traces. An eval without trace context is a dead end. My recommended baseline is an execution trace tree. See AI agents and agent orchestration.

Start with behavior, not implementation (executable specifications)

LLM systems make it easy to overfit to implementation details:

• “The agent must call search_docs before answer_user.”
• “The prompt must include this exact bullet list.”

That’s brittle. It breaks the second you change model, tool schema, or routing.

Specify behavior instead:

• The output must be valid JSON matching a schema.
• The agent must not take an irreversible action without approval.
• For RAG, citations must reference retrieved documents.

Executable specs are the durable artifact. Alger’s “verification bottleneck” point isn’t some abstract philosophy. It’s day-to-day operations.

One concrete pattern I like is two-layer checks:

• Hard checks (must-pass): JSON schema, forbidden actions, PII redaction rules.
• Soft checks (scored): semantic correctness, completeness, tone.

Hard checks gate merges. Soft checks inform decisions and trend monitoring.

This is also where AI security meets testing. You can encode real policies as tests. Not vibes.

Metamorphic tests for LLMs (relations that survive model churn)

Metamorphic testing is the most underused technique in LLM evals.

Metamorphic tests don’t assert “this exact output.” They assert relations that should hold when you transform the input.

You can think of it like property-based testing for language.

Metamorphic relations that actually work

Here’s my short catalog by task type. These are designed to be useful in CI, not clever in a research paper.

Classification

• Paraphrase invariance: paraphrase the input, the class should not change.
• Negation flip: add explicit negation and the label should flip (for binary sentiment/toxicity-style tasks).
• Irrelevant detail invariance: add an unrelated sentence. Label stays the same.

Extraction

• Format preservation: changing surrounding prose should not change extracted fields.
• Order invariance: permuting a list in the input should permute outputs or keep set-equivalence.
• Null sensitivity: remove a field. The extracted value must become null, not invented.

Summarization

• Length monotonicity: with a tighter word limit (e.g., 200→100 words), summary must not introduce new facts.
• Citation consistency: if you require citations, cited facts must exist in the source.

RAG (retrieval-augmented generation)

• More relevant context should not hurt: adding a highly relevant chunk should not decrease correctness.
• Contradiction handling: if two retrieved docs disagree, the model must surface uncertainty or follow a policy.

Tool use / agents

• Deterministic tool replay: given the same tool outputs, the final answer should be stable.
• Permission monotonicity: tightening permissions must not produce “phantom success.” It should fail loudly.

Metamorphic testing is especially strong under model churn because you’re not overfitting to a single “golden answer.” You’re pinning down invariants.

If you want to go deeper on RAG-specific failure modes, my go-to companion post is RAG.

Reducing variance: what you can and cannot control

Before you design tolerance bands, reduce variance you don’t need.

What you can control (usually)

• Sampling parameters: set temperature and top_p. For gating, I often use temperature=0 or 0.2.
• Seed control: some APIs support a seed. If you have it, record it. If you don’t, accept that.
• Tool determinism: mock time, freeze external APIs, stub flaky endpoints.
• Retrieval snapshotting: pin an index build ID. Don’t eval against a moving target.
• System prompt versioning: hash it, store it, diff it.

What you can’t control (stop pretending)

• Silent vendor backend changes.
• Model routing and mixture-of-experts decisions.
• Latency-driven timeouts that cascade into tool behavior.

The fix is observability plus cohort comparison, not denial.

If you’re doing local LLM evals, you get more control (weights are pinned), but you still have nondeterminism from kernels, quantization differences, and tool timing. Different source of variance. Same problem.

Designing stochastic tolerance bands (a recipe that doesn’t create flaky CI)

Most advice hand-waves this part, which is wild because this is the part you’ll be living with.

A tolerance band is a pass/fail rule that accepts expected stochastic variation while rejecting real regressions.

Step 1: Decide what you’re gating

Gate on behavioral success rate, not on average score.

Example: “At least 95% of cases must pass hard checks.”

For a semantic judge score (0–5), convert it into a binary outcome with a rubric threshold. Example: “score ≥ 4 counts as pass.”

Step 2: Pick N runs based on variance, not vibes

I use:

• N=3 for smoke
• N=5 for PR gate
• N=10 for nightly

If you’re running an agent with tools and retries, start at N=5. If you’re evaluating a pure prompt, N=3 is usually enough.

Step 3: Use a pass-rate threshold + confidence guard

A simple, robust rule:

• For each case, run N times.
• A case passes if it passes at least k times.
• The suite passes if at least P% of cases pass.

Concrete defaults I’ve used successfully:

• N=5, k=4 (80% per-case)
• Suite threshold P=95%

This rejects flaky behavior while allowing rare sampling weirdness.

If you want a statistical framing: treat each case as a Bernoulli trial and compute a confidence interval for pass rate. You don’t need fancy math in CI. You do need consistency in how you set thresholds.

Step 4: Make it fail-closed

Raju’s “fail-closed gate” point matters.

If a case didn’t run, that is not a pass.

• Missing artifacts should fail.
• All-skipped suites should fail.
• Judge model outage should fail (or fall back to human review explicitly).

If you’re serious about AI in production, your gate cannot silently degrade.

Step 5: Put a cost budget on it

A gate that costs $200 per PR will be bypassed. A gate that costs $2 per PR will be used.

Set a budget like:

• Smoke lane: < 2 minutes, < $0.50
• PR gate: < 10 minutes, < $5
• Nightly: < 60 minutes, < $50

Then enforce it like any other reliability SLO.

Based on the benchmark data I maintain at https://www.kunalganglani.com/llm-benchmarks, model latency and throughput vary wildly across providers and model sizes. That variance directly affects how many runs you can afford in CI before teams start skipping gates.

Don’t let the student grade the exam (judge-model pitfalls)

Using an LLM to judge an LLM isn’t automatically wrong. It’s just easy to do in a way that produces fake confidence.

Alger calls out the “student grading the exam” failure mode. If the same model generates and judges, you can get correlated errors and a suspiciously stable score.

Sergei Parfenov shows an adjacent failure mode: AI-generated tests can be weak or mis-specified, and they can make coding agents look better or worse. The lesson generalizes. Test the tests. Read him here: Sergei Parfenov.

How I make LLM-as-judge less sketchy

• Use a different model family as judge when possible.
• Freeze the judge prompt and rubric. Version it.
• Calibrate: sample 20–50 cases monthly for human spot-checks.
• Measure inter-rater drift: if your judge score distribution shifts by > 10% week-over-week, investigate.

Also, if the judged output includes citations, validate citations with deterministic code.

For security evals, don’t rely on a judge at all. Use explicit policies and deterministic checks, especially for LLM security and prompt injection.

Turn credible evidence into a fail-closed gate (CI/CD structure)

If you try to run everything on every PR, you’ll either go broke or everyone will turn it off.

Here’s the structure that’s worked for me and for teams I respect:

1. Pre-merge smoke: 10–20 named cases, hard checks only, N=3. Fail-closed.
2. Pre-merge gate: 50–100 cases, hard + key soft checks, N=5. Fail-closed.
3. Post-merge canary: run on a small slice of production traffic, compare cohorts daily.
4. Nightly regression: big suite, N=10, includes metamorphic and adversarial cases.

Treat this like CI/CD for AI. Fast lane prevents obvious breakage. Slow lane catches drift.

If you’re already doing agent failure testing, connect the harnesses. Tool failures are a major source of nondeterminism. See AI agents and production AI.

Closing the loop: triage eval regressions + postmortem template

When an eval regresses, the first argument is always “the model got worse.” That’s usually the wrong first guess.

I use a simple taxonomy:

• Model change: vendor upgrade, routing change.
• Prompt change: system prompt edit, prompt compression, template refactor.
• Tool change: tool schema change, timeout behavior, retries.
• Retrieval change: index refresh, embedding model change, chunking change.
• Harness bug: incorrect labels, flaky judge, broken fixtures.

Triage workflow (what I do in practice)

1. Re-run the failing named cases 10 times with identical metadata.
2. Replay with tools mocked (same tool outputs). If failure disappears, it’s tool variance.
3. Pin retrieval snapshot. If failure disappears, it’s retrieval drift.
4. Swap model backends (if you can). If failure persists, it’s your prompt or harness.
5. If it’s judge-related, run a human spot-check on 10 samples.

Eval regression postmortem template

Use this when a regression escaped, or when the gate got flaky enough that people bypassed it.

• Incident ID / date:
• What regressed: metric name, threshold, baseline vs candidate numbers (e.g., 96%→89%)
• Detection: where it was caught (PR gate, nightly, production canary), timestamp
• Blast radius: % traffic affected, which features/users, duration
• Root cause category: model / prompt / tool / retrieval / harness
• Contributing factors: missing metadata, unversioned prompts, judge drift
• Why didn’t the gate catch it sooner: suite coverage gap, tolerance band too wide, smoke too small
• Fix shipped: prompt/tool/retrieval/harness changes
• Preventive actions: add named cases (count), add metamorphic relation, tighten thresholds, add monitoring
• Follow-ups: owner + due date

I’m opinionated here: write the postmortem even if it’s “just an eval.” Your eval system is production infrastructure. Treat it with the same seriousness as a flaky deploy pipeline.

If you want to see what “closing the loop” looks like for security incidents, the principles are the same. Start from AI security.

Here’s my prediction for 2027: orgs will stop asking “which model is best?” and start asking “which eval harness do we trust?” If your harness isn’t versioned, cohort-based, and built for churn, you’re not shipping AI. You’re shipping demos.

Photo by Kashifa Sharif on Unsplash.