AI Agent Observability Logging Schema [2026]: OTel + Redaction

AI agent observability logging schema is the difference between “we shipped an agent” and “we can debug it at 2 a.m. without leaking customer data.” I’ve watched too many teams celebrate the demo and then faceplant the first time the agent starts looping, burning tokens, and timing out in production.

Here’s the nasty part: the most tempting thing to log (raw prompts and tool payloads) is also the easiest way to spray secrets into your telemetry pipeline. If you’re not treating your logs as a data exfiltration channel, you’re doing security theater.

Key takeaways

• Treat an agent run like a distributed trace. One trace per run, spans for planning/LLM/tool/retrieval steps, and stable IDs everywhere.
• Default-on production logging should be metadata-only. Full prompt/tool content must be opt-in, short-lived, access-controlled, and aggressively redacted.
• Tool I/O logging should be allowlist-first, with redaction transforms in the SDK and again in the collector (defense in depth).
• Sampling for agents should be head-based by default, with tail sampling triggered by failures, retries, high cost, and “stuck loop” signatures.
• If you align your fields to gen_ai.* and mcp.* now, you avoid migration churn as OpenTelemetry GenAI conventions harden.

If your agent is a distributed system, your observability has to be one too.

What is LLM Observability?

LLM observability is the practice of instrumenting large language model-backed systems so you can understand behavior, performance, quality, and cost in production.

For agents, it’s not “prompt + response.” That mental model is how you end up with a pile of useless logs and a compliance incident. An agent run is a sequence of decisions: planning, tool calls, retrieval, retries, policy checks, and some final outcome you can grade.

This matters more in 2026 than it did even a year ago. OpenTelemetry’s GenAI semantic conventions now live in a dedicated repository, open-telemetry/semantic-conventions-genai. They include explicit agent spans, events, and Model Context Protocol (MCP) context propagation guidance. That’s where the ecosystem is going. If your telemetry vocabulary drifts from it now, you’re signing yourself up for a migration later. Migration taxes are real. I hate them.

Security-wise, OWASP’s GenAI LLM Top 10 2026 dropped on August 4, 2026, and the project has grown to 600+ contributing experts across 18+ countries with nearly 8,000 active community members. Translation: prompt/data leakage is no longer a “maybe.” It’s a thing that happens to real companies, on real timelines, with real consequences.

→ Related: OpenTelemetry Instrumentation for AI Agents [2026]: Ship It

Traditional vs. LLM Observability: What’s the Difference?

Traditional observability assumes requests are mostly deterministic. Same inputs, mostly same outputs. If something breaks, traces and logs usually tell a reasonably linear story.

Agents don’t.

• Nondeterminism is normal. Temperature, tool timing, retrieval variance, model drift. Two “identical” runs can diverge in annoying ways.
• Control flow is dynamic. Agents decide what to do next. One user request can fan out into 3 tool calls or 30.
• Payloads are radioactive. Prompts contain user data, internal instructions, credentials you didn’t know were there, and compliance landmines.
• Cost is a first-class metric. Token usage and tool spend can spike silently with retries, self-corrections, or loops.

So yes, you still need traces, metrics, and logs. But you need a schema that assumes the run is messy, and defaults to not leaking sensitive data.

The Five Pillars of LLM Observability

I like the “five pillars” framing because it forces you to stop obsessing over whether you captured the prompt, and start measuring what actually matters.

1. Reliability debugging: where runs fail, where they loop, which tool calls are flaky.
2. Latency: not just end-to-end. Step-level breakdowns, especially retrieval and tools.
3. Quality: outcomes, rubric scores, user feedback, automated checks.
4. Cost: tokens, retries, tool spend, per-tenant attribution.
5. Security & compliance: redaction, access control, retention tiers, auditability.

If you only do the first three, you’ll ship faster. If you ignore the last two, you’ll eventually have an incident. Not because you’re unlucky. Because your logging pipeline becomes your leak.

Getting Started: the schema I’d ship in a weekend

You can implement this in two layers:

• Application logging contract: JSON events you emit from your agent runtime
• OpenTelemetry mapping: trace/span/attribute conventions that make it portable across vendors

The point isn’t “pick the perfect vendor.” The point is you have a contract your teams follow, and you can move backends without rewriting your entire debugging story.

The schema levels

Think in four levels:

• Run: the whole agent invocation (one trace)
• Step: planning / reasoning / decide-next-action (spans)
• Tool call: external side effects (spans)
• Content blobs: raw prompt/response/tool payloads (almost never in spans)

Agent observability logging schema table

This is the minimum viable contract. Copy it into your internal docs and make teams conform to it.

A few numbers to make this concrete:

• `ttl_hours`: I default to 24 hours in prod for content blobs. Debuggability without building a compliance nightmare.
• `top_k`: log it when you do retrieval. I’ve seen RAG regressions caused by a quiet shift from 5 to 20 retrieved chunks.
• `loop.iteration`: if your agent can loop, you need a counter. An iteration cap of 8–12 is a sane default for many workflows.

What NOT to log (and what to log instead)

This is where most teams screw it up.

Don’t log:

• System prompts verbatim
• Chain-of-thought / private reasoning
• Raw user content by default
• Raw tool payloads by default (especially anything that can contain credentials)

Log instead:

• prompt_template_id and prompt_template_version
• prompt_hash (hash of the rendered prompt after deterministic normalization)
• input_classification (e.g., public, internal, restricted)
• tool.payload_schema_version plus allowlisted fields

The goal is to make prompts diffable without making them readable.

Set up tracing with OpenTelemetry (minimal but correct)

OpenTelemetry gives you a portable shape: trace → spans → events → attributes. The GenAI semantic conventions extend that shape with vocabulary that’s actually relevant to model calls and agent steps.

Authoritative baseline: the OpenTelemetry community specification.

Trace boundaries

• One trace per agent run.
• Root span name: agent.run {agent.name} (internal naming). Map to GenAI conventions inside.

Span mapping to GenAI conventions

OpenTelemetry’s GenAI span guidance includes:

• Agent spans like plan and execute tool spans in the agent spec. See the plan and execute tool sections in the OpenTelemetry community.
• Client inference spans for model calls with gen_ai.operation.name and gen_ai.request.model. See OpenTelemetry community.

My minimal mapping:

• Root: agent.run (INTERNAL)
• Plan span: plan (INTERNAL)
• LLM inference span: {gen_ai.operation.name} {gen_ai.request.model} (CLIENT or INTERNAL)
• Execute tool span: execute_tool {tool.name} (CLIENT if remote)
• Retrieval span: retrieval {retrieval.source} (INTERNAL)

Attributes you should standardize

At minimum:

• Identity: agent.name, agent.version, environment, tenant_id
• Model: gen_ai.provider.name, gen_ai.request.model, gen_ai.operation.name
• Tokens: tokens.in, tokens.out, tokens.total
• Cost: cost.usd at span-level and run.cost.usd at root
• Control flow: retry.count, loop.iteration

Cost note: if you can’t attribute spend per tenant and per user, you don’t have “cost observability.” You have vibes.

Events vs. attributes for payloads

The GenAI semantic conventions discuss strategies for capturing instructions/inputs/outputs: full buffered content, attributes, external storage, or streaming chunks.

The boring answer is the right one: metadata in attributes; content in external blobs.

Use span attributes for:

• hashes
• lengths
• MIME types
• schema versions

Use events only when:

• you’re in a controlled dev environment
• you have strict access controls
• you need just enough content for a short time

And even then, redact first.

How do you trace tool calls in an LLM agent?

Tool calls are where your agent stops being “chat” and starts being “software.” They’re also where you leak secrets if you log like it’s 2015.

A practical approach:

1. Create a tool span for every tool invocation.
2. Put timing, status, and a stable request ID on the span.
3. For request/response payloads, log only allowlisted fields (think: resource_id, query_type, result_count).
4. Store full payloads as encrypted blobs with TTL, and only when explicitly enabled.

If you’re using MCP tools/servers, OpenTelemetry recommends context propagation by injecting traceparent, tracestate, and baggage into params._meta and extracting it on the receiver as the remote parent. See the MCP conventions doc by the OpenTelemetry community.

That one detail is what makes your tool spans line up cleanly across process boundaries.

How do I avoid logging secrets and PII in LLM prompts and tool outputs?

Treat telemetry as a data exfil path. Because it is.

Here’s the defense-in-depth model that actually holds up under reality (meaning: under deadlines, new services, and humans making mistakes).

1) Allowlist-first logging

For each tool, define:

• tool.payload.allowlist.paths (JSONPaths you permit)
• tool.payload.denylist.paths (high-risk fields you always drop)

Assume “log nothing” unless explicitly allowed.

2) Redaction transforms (SDK-side)

Before anything leaves the process:

• redact credentials (Authorization, api_key, token, cookie)
• redact common PII patterns (emails, phone numbers)
• truncate long strings (cap at 256 chars for any field)
• hash stable identifiers (SHA-256) when you need correlation without disclosure

3) Redaction transforms again (collector-side)

Someone will bypass your SDK. Or a new service will ship without it. So you do it again in the OpenTelemetry Collector:

• drop known sensitive attributes
• enforce max attribute size
• enforce environment-based policies (prod stricter than dev)

4) Don’t put secrets in prompts

This sounds obvious until you see real systems.

Secrets belong in a secret manager and should never be embedded into prompts. The authoritative baseline is Google Cloud: minimize access, rotate, audit, and never leak via logs.

What are spans and traces in OpenTelemetry?

• A trace is a tree representing one end-to-end operation.
• A span is one timed unit of work within that trace.

For agents:

• trace = one run
• spans = planning step, model call, retrieval, tool execution, policy check
• logs/events = details you might want to attach (carefully)

If you adopt that mental model, most schema decisions stop being mysterious.

What is tail-based sampling and when should I use it?

Head-based sampling decides at the start. Tail-based sampling decides after the trace finishes, based on what happened.

For agents, tail sampling is ridiculously effective because “interesting” traces are rare but expensive:

• failures
• high retry count (>= 2)
• loop iterations above a threshold (>= 6)
• high cost (e.g., run.cost.usd > $0.25 for consumer workflows)
• policy violations

Default: head sample low (say 1–5%) for volume control.

Then: tail sample 100% of “bad” runs so you can debug what matters without setting money on fire.

How do you correlate logs with traces for debugging?

Use three IDs everywhere:

• trace_id (OTel)
• run_id (your business identifier)
• request_id (edge ingress)

Logs should include trace_id and span_id. Your agent events should include run_id. This lets you start from a user complaint, find the run, then pivot into traces.

And yes, this is why “one trace per run” is such a powerful simplification.

Trace a RAG application (agent + retrieval)

RAG (Retrieval-Augmented Generation) is where observability gets subtle.

You need to see:

• retrieval latency
• retrieval count (top_k)
• which index/collection you hit
• reranker model (if any)
• how many tokens you spent stuffing context

But you do not want raw retrieved chunks in traces.

Instead:

• log doc_id and doc_hash
• log chunk_id and chunk_hash
• log retrieval.query_hash

Then store actual content only behind explicit gates.

If you want a deeper security playbook here, I’ve already written the adjacent piece on RAG.

View traces (what “good” looks like)

When you open a trace view for a run, you should be able to answer in under 60 seconds:

• Did it succeed?
• Where did time go?
• Where did cost go?
• Which tool failed?
• Did it loop?
• Was a policy gate triggered?

If you can’t, you’re collecting trivia, not observability.

For deeper control-flow patterns (retries, checkpoints, HITL), see AI agents and my guide on AI in production.

Monitor performance (latency + cost)

Monitoring agents is mostly about catching regressions before your users do:

• P95 step latency changes
• token usage drift
• tool error rate spikes
• loop frequency

I keep two dashboards:

1. Run-level SLOs: success rate, p95 end-to-end latency, p95 cost.
2. Step-level heatmap: which spans are dominating time/cost.

If you want a measurement discipline for LLM latency, I wrote up the methodology I use in LLM latency.

For cost math (including retries and tools), connect this with LLM cost and the practical breakdown in AI in production.

Default-on vs opt-in logging in production for agents

Here’s my stance: production should be metadata-only by default. If you need content, you earn it with explicit controls.

Default-on (prod):

• spans with timings + statuses
• token counts
• cost estimates
• tool names + status codes
• prompt/template IDs + hashes

Opt-in (prod, time-limited):

• full prompts/responses
• full tool payloads
• retrieved content

And when you enable opt-in, it should be:

• scoped to a tenant/user/session
• time-boxed (e.g., 1 hour window)
• stored as encrypted blobs with TTL

How to store full prompt/response safely (without bloating traces)

The pattern that scales:

• Put raw content in object storage (S3/R2/GCS).
• Make it content-addressed (sha256 as key).
• Encrypt with KMS.
• Apply TTL lifecycle policy.
• Store only blob_id plus content_sha256 plus ttl_hours in your traces.

This keeps traces lightweight and searchable, and keeps sensitive data behind a stricter access model than “anyone with read access to logs.”

Retries, self-corrections, and agent loops: how to represent them

A few rules:

• A single logical model call with retries should be one inference span, with retry.count and retry events.
• Agent loops should increment loop.iteration and emit a loop span per iteration.
• If the agent is stuck (same tool call repeated), emit a stuck.signature_hash so you can alert on it.

Once you do this, tail sampling becomes mechanical. “Keep any trace where loop.iteration >= 6.” Easy.

How to design a redaction pipeline (SDK + collector)

This is the part I want teams to stop hand-waving.

1. SDK-side: redact early so secrets never leave process memory.
2. Collector-side: redact again because someone will ship without the SDK.
3. Backend-side: enforce access control and retention policies.

I’ve built the 7-agent blog publishing pipeline for this site, and one of the biggest lessons was boring: deterministic gates catch issues earlier than “we’ll review it later.” Same idea here. Put deterministic redaction and drop rules in front of your exporter, not as a best-effort afterthought.

To go deeper on agent security failure modes, connect this to my broader AI security and the specific risk of prompt injection.

Internal links you’ll likely want next

• AI agents
• AI in production
• agentic AI
• agent orchestration
• OpenTelemetry
• RAG
• prompt injection
• Claude Code
• local LLM

Photo by Luca Bravo on Unsplash.