# How to Use MLflow LLM Evaluation Tracing [2026] (Spans → Gates) > Turn MLflow traces into living eval datasets, rubric scorecards, and CI/CD regression gates for tool-calling agents. Stop debugging with random logs. - Canonical: https://www.kunalganglani.com/blog/mlflow-evaluation-tracing-gates - Author: Kunal Ganglani - Published: 2026-09-23 · Updated: 2026-09-23 - Category: AI and Machine Learning · Tags: mlflow, evals, llmops, observability, ai-agents ## TL;DR When you ship a tool‑calling AI assistant, the hard part isn’t getting it to work once. It’s proving it still works after every prompt tweak, model change, or tool update. MLflow tracing can record each user interaction as a trace made of smaller steps (spans), including tool calls. Those spans already contain the inputs, outputs, and failures you need for testing. This guide shows how to turn MLflow traces into an evaluation dataset, score it with a simple rubric, track the results over time, and block releases when quality drops. The takeaway: use observability data to power your evals, not a stale spreadsheet. ## How to Use MLflow LLM Evaluation Tracing [2026] (Spans → Gates) You can go from “we have traces” to “we have regression gates” in **60–90 minutes**. Not a research project. Not a quarter-long platform rebuild. Just a tight loop: **MLflow tracing → eval dataset → scorecard → CI gate**. ![Nvidia logo on a green background with abstract spheres](https://cdn.sanity.io/images/vzekdneq/production/dbeab446066aa6f90fb1bbf620a492a8623506e7-1200x675.webp) If you’re searching for **mlflow llm evaluation tracing**, here’s the workflow that actually holds up in 2026: stop treating observability and evals like two separate hobbies owned by two separate people. Your production traces are already the highest-signal dataset you’ve got. The only move is turning spans into rows. Here’s the pipeline I’d ship for a tool-calling agent: 1. Instrument your agent so every tool call becomes a span. 1. Store traces in MLflow. 1. Extract a balanced sample of spans into an eval dataset. 1. Score it with a rubric, track the scorecard in MLflow, and compare versions. 1. Fail CI (or block a canary) when quality regresses. This is one of those things where the boring answer is actually the right one. Your eval set should not be a sad spreadsheet from two quarters ago. It should be your product. [IMAGE: section-break] ## What is MLflow Tracing? **MLflow Tracing is MLflow’s observability feature for LLM and agent workflows that records requests as traces made up of spans (e.g., model calls and tool calls) so you can inspect behavior, latency, errors, and metadata in a UI and query it later for analysis and evaluation.** ![Two nvidia titan x graphics cards side by side](https://cdn.sanity.io/images/vzekdneq/production/b6e62bfa053ea230e29467b971c39f8af28e7a26-1200x675.webp) A few definitions you need to keep straight (because teams get sloppy here and then wonder why they can’t build gates): - **Trace**: one end-to-end user interaction (for an agent: plan → tool calls → final response). One trace typically maps to one “task.” - **Span**: a timed step within the trace. For LLM apps, useful spans include `llm`, `tool`, `retrieval`, `rerank`, `guardrail`, and `postprocess`. - **Tool-call span**: a span that wraps a function/tool invocation. This is the best unit for turning production traffic into eval rows because it has inputs, outputs, and a concrete notion of success/failure. If you want the official “what does MLflow show me” orientation, the walkthrough by [Daniel Liden](https://www.youtube.com/watch?v=X5P-zvGGE0g) (Developer Advocate at Databricks) is the quickest way to see it end-to-end. The point of this post is simple: **spans are already structured test cases**. Most teams just refuse to treat them that way. Concrete math: if your agent calls ~3 tools on average and you do **10,000** interactions/day, that’s ~**30,000 tool-call spans/day**. Sample **0.5%** and you get **150** high-signal eval rows per day without anybody writing synthetic prompts or arguing about “representative scenarios” in a meeting. [IMAGE: section-break] ## Capture tool calls as spans (and the attributes that matter) Tool calls are where agents fail in ways normal chat apps don’t: wrong arguments, wrong tool choice, half-baked retries, timeouts, and the classic “tool succeeded but the model misused the output.” ![Nvidia logo on a green background with abstract 3D elements](https://cdn.sanity.io/images/vzekdneq/production/f40384b138d01ca47aa0d4cddccc3cefafeab306-1200x675.webp) If you’re building [AI agents](/pillars/ai-agents) in production, instrumenting tool calls as first-class spans is non-negotiable. ### Minimal OpenTelemetry-style schema for tool-call spans You can log a hundred attributes. You shouldn’t. Start with a minimal schema you can keep stable for a year, and add fields only when you can answer the question “what scorecard metric will this unlock?” I like aligning with OpenTelemetry-style conventions so you don’t paint yourself into a corner. The closest thing to a community reference right now is the OpenInference semantic conventions maintained by [Arize AI](https://github.com/Arize-ai/openinference). Here’s a minimal mapping that’s evaluation-friendly and won’t fight the direction standards are heading. | Concept | Span field | Example | Why it matters for evals | | --- | --- | --- | --- | | Trace ID | `trace_id` | `9f2c…` | Join score → trace drill-down | | Span ID | `span_id` | `a13b…` | Uniqueness per step | | Parent span | `parent_span_id` | `root` | Reconstruct agent tree | | Span kind | `span.kind` | `tool` | Filtering/slicing | | Tool name | `tool.name` | `calendar.create_event` | Slice evals by tool | | Tool args (redacted) | `tool.arguments` | `{ "start": "2026-09-23" }` | Argument correctness | | Tool result (redacted) | `tool.result` | `{ "event_id": "…" }` | Ground truth / validation | | Tool status | `tool.status` | `ok` / `error` / `timeout` | Reliability metrics | | Retries | `tool.retries` | `2` | Regression signal | | Model name | `llm.model` | `gpt-4.1` | Compare model variants | | Prompt hash | `prompt.hash` | `sha256:…` | Detect drift without leaking text | | User segment | `user.segment` | `free` / `enterprise` | Bias + representativeness | | Latency ms | `latency_ms` | `1830` | Correlate quality with perf | | Cost (optional) | `cost.usd` | `0.0042` | Quality per dollar | Even if you only log **12–15 attributes**, you can build 90% of the scorecards you actually need. ### My strong opinion on what *not* to log - Don’t log raw prompts by default. Hash them, and put raw text behind strict access controls. - Don’t log tool outputs that might contain secrets or PII unless you have a redaction pipeline. If you need a practical privacy posture, start with [LLM data leakage](/blog/llm-data-leakage-playbook) patterns and treat traces like production logs. Because that’s what they are. [IMAGE: section-break] ## Transform spans into an evaluation dataset (rows you can score) You have two sane granularities: 1. **One row per interaction (trace-level)**: good for “did the user get the right final answer?” 1. **One row per tool call (span-level)**: good for “did the agent choose and use tools correctly?” For agent systems, I prefer span-level datasets. That’s where regressions hide. ### A practical dataset schema Create a table (Delta/Parquet/CSV, doesn’t matter at first) with columns like: - `trace_id` - `span_id` - `timestamp` - `tool_name` - `tool_arguments_redacted` - `tool_result_redacted` - `tool_status` - `final_response_redacted` (optional but useful) - `expected_tool` (optional “ground truth” if you have rules) - `expected_properties` (optional: invariants like “event duration > 0”) - `metadata` (segment, release, model, prompt hash) Sampling is where people accidentally lie to themselves. If you have **8 tools**, do not sample uniformly across all spans. You’ll over-sample the popular tool and barely see the ones that break once a day and ruin your on-call. A dead-simple balanced sampler: - **N = 50** spans per tool per day (cap at availability) - plus **all** spans with `tool.status != ok` - plus **top 20** slowest spans per tool (latency tail) Now you’ve got an eval set that’s representative enough to track overall quality, and failure-heavy enough to catch regressions early. ### Avoid data leakage and prompt injection artifacts in your eval corpus If you replay production traffic into evals, assume it contains: - secrets (yes, people paste API keys into chat) - PII - adversarial strings (prompt injection attempts) My baseline: - redact obvious patterns (emails, SSNs, tokens) - keep an allowlist of fields that are allowed into eval rows - store raw traces with tighter retention than derived eval rows If you’re actively testing [prompt injection](/blog/prompt-injection-regression-testing-ci), keep adversarial samples in a separate eval suite. Don’t contaminate your “normal traffic” scorecard and then wonder why the chart is screaming every day. [IMAGE: section-break] ## Design a rubric scorecard and run evals repeatedly Most teams screw this up by trying to grade “quality” as one number. You get a clean dashboard and learn nothing. For tool-calling agents, I want a scorecard with at least **5** criteria: 1. **Tool selection correctness** (did it call the right tool?) 1. **Argument correctness** (did it pass valid args?) 1. **Tool outcome handling** (did it interpret results correctly?) 1. **User-facing correctness** (is the final response correct?) 1. **Safety/compliance** (did it avoid disallowed actions?) Use a simple scale like **0/1/2** (fail/partial/pass). It’s boring. It works. Grader options: - deterministic checks (JSON schema validation, invariants) - reference comparisons (known-good tool output) - LLM-as-judge with a tight rubric and calibration LLM-as-judge isn’t perfect. I still use it for one reason: when you calibrate it on a small labeled set and keep it stable, it’s a great regression detector. This connects directly to work I’ve done running this site’s multi-agent publishing pipeline. After shipping a deterministic SEO quality gate for **261+ posts** on kunalganglani.com, I learned the same lesson applies here: **deterministic gates catch more regressions than “just use a bigger model to review it.”** That’s not theory. It’s straight out of the incident log from operating the pipeline week after week. Track your scorecard runs in MLflow like you would any other experiment: - one MLflow experiment per app (or per agent) - tag runs with `git_sha`, `prompt_hash`, `model`, `toolset_version` - log metrics per slice: by `tool.name`, by `user.segment`, by `error_type` Concrete metrics I’d compute every run: - overall mean score (0–2) - **p10** score (tail quality) - tool error rate (%) - tool timeout rate (%) - median latency (ms) and **p95 latency (ms)** If p95 is exploding while quality is flat, you still shipped a regression. Users feel latency more than your dashboard feels smug. I’ve written more about that tradeoff in [LLM observability metrics](/blog/llm-observability-metrics) and [AI agent latency budgets](/blog/ai-agent-latency-optimization-budget). Here’s the official walkthrough to anchor what MLflow tracing looks like in practice: [Watch: MLflow Tracing: Introduction & Tutorial](https://www.youtube.com/watch?v=X5P-zvGGE0g) [IMAGE: section-break] ## Turn scorecards into regression gates (CI/CD and canaries) If your evals don’t block merges or canary promotions, they’re just charts. I like two gate types. ### Gate 1: CI regression gate (fast, deterministic, cheap) Run a small replay suite on every PR: - **N = 200** tool-call rows - stratified by tool - includes last week’s top failure clusters Fail the build if: - any tool’s mean score drops by **> 0.1** on a 0–2 scale - overall mean drops by **> 0.05** - tool timeout rate increases by **> 1%** absolute Those numbers aren’t magic. They’re deliberately annoying. They force you to define what “good” means and stop hand-waving. If you already have a broader evaluation harness, plug this into it. My references for thinking clearly about flaky systems are [non-deterministic AI system testing](/blog/non-deterministic-ai-testing) and [AI engineering evals gates](/blog/ai-engineering-evals-gates). ### Gate 2: Canary gate (slice-based, production-real) For canaries, compare two live versions on the same traffic slices: - tool selection score by tool - timeout rate - p95 latency Block promotion if any critical slice regresses, even if the overall average looks fine. The classic failure mode is “enterprise calendar got worse, but the free-tier FAQ bot got better, so the mean stayed flat.” Congrats. You just broke the customers who pay you. If you’re shipping [production AI](/pillars/ai-engineering-production), slice-based gates are the difference between “we have evals” and “we can control releases.” [IMAGE: section-break] ## Dashboards: link quality regressions to trace drill-down The dashboard layout I’ve seen work is three panes: 1. **Scorecard overview**: overall score + per-criterion trends (daily/weekly) 1. **Reliability & performance**: tool error rate, timeout rate, retries, p95 latency 1. **Slice explorer**: tool, model, prompt hash, user segment, release The drill-down needs to be one click: - “Tool selection score dropped for `calendar.create_event` by **0.18** today” - click → list of worst rows - click → MLflow trace view for that row - inspect the exact tool arguments and model output If you can’t get from a regression chart to an exact trace in under **30 seconds**, your system will rot into “debug by vibe.” And then your team will start arguing about prompts like it’s astrology. I’m also a fan of a daily computed “Top 10 regressions” board: - worst tool slice by delta - worst user segment by delta - worst prompt hash by delta If you want a broader, vendor-neutral approach, pair this with [OpenTelemetry instrumentation for AI agents](/blog/opentelemetry-ai-agents-instrumentation) and [vendor-neutral LLM observability](/blog/llm-observability-vendor-neutral). [IMAGE: section-break] ## Sampling and curation: build an eval set that stays representative This is where teams accidentally sabotage themselves. If you always sample “interesting failures,” your scorecard becomes a panic dashboard. If you only sample “random traffic,” regressions hide in the tails. I like a three-bucket eval dataset: - **Baseline bucket (60%)**: stratified random sample by tool and segment - **Failure bucket (30%)**: all non-OK spans + top retry spans - **Edge bucket (10%)**: long-context, long-latency, weird-user-input cases Then freeze snapshots: - daily rolling set (freshness) - weekly frozen set (comparability) - monthly “golden set” (long-term trend) Freshness matters because MLflow tracing is still evolving and the ecosystem is converging on common conventions. OpenInference is a good example of where things are going. But you still want your own minimal schema to stay stable. One more practical warning: if you’re evaluating tool calls that can write data, build a safe replay mode. Teams have absolutely replayed “delete user” operations because someone treated eval replay like unit tests. Don’t. If your agent has dangerous tools, use approval patterns. Start with [tool approval patterns](/blog/tool-approval-patterns-ai-agents) and apply the same mindset as [MCP server security best practices](/blog/mcp-server-security-best-practices). ## The part I think will matter next Within a year, “LLMOps” is going to collapse into the same mental model as normal software delivery: **instrumentation → dataset → tests → gates**. The teams who win won’t be the ones with the fanciest agent framework. They’ll be the ones who can point at a trace, point at a score, and say: this failed, here’s why, and we can prove we fixed it. If you’re building tool-calling agents and you still don’t have a trace-derived eval dataset, stop adding features for a week and wire this up. Seriously. That’s the week you buy back the next six months. Photo by Stephen Dawson on Unsplash.