# AI Agent Observability Logging Schema [2026]: OTel + Redaction

> A copy‑paste JSON logging contract for agent runs (spans, tool I/O, cost, retries) plus a minimal OpenTelemetry mapping you can implement in a weekend—without leaking secrets.

- Canonical: https://www.kunalganglani.com/blog/ai-agent-observability-logging-schema
- Author: Kunal Ganglani
- Published: 2026-08-10 · Updated: 2026-08-10
- Category: AI and Machine Learning · Tags: llmops, observability, opentelemetry, ai-agents, security

## TL;DR

AI agents are hard to debug because they don’t always fail the same way twice, and the obvious fix—logging prompts and tool payloads—can leak secrets. A safer approach is to log only metadata by default: timings, statuses, token counts, costs, retries, and stable IDs. Store full prompt/tool content separately as encrypted files with a short expiry, and only turn it on when you’re actively debugging. Treat each agent run like a trace with spans for planning, model calls, retrieval, and tool execution, and use sampling that keeps every failed or expensive run. The result is observability without accidental data spills.

AI agent observability logging schema is the difference between “we shipped an agent” and “we can debug it at 2 a.m. without leaking customer data.” I’ve watched too many teams celebrate the demo and then faceplant the first time the agent starts looping, burning tokens, and timing out in production.

Here’s the nasty part: the most tempting thing to log (raw prompts and tool payloads) is also the easiest way to spray secrets into your telemetry pipeline. If you’re not treating your logs as a data exfiltration channel, you’re doing security theater.

**Key takeaways**

- Treat an agent run like a distributed trace. One trace per run, spans for planning/LLM/tool/retrieval steps, and stable IDs everywhere.
- Default-on production logging should be metadata-only. Full prompt/tool content must be opt-in, short-lived, access-controlled, and aggressively redacted.
- Tool I/O logging should be allowlist-first, with redaction transforms in the SDK and again in the collector (defense in depth).
- Sampling for agents should be head-based by default, with tail sampling triggered by failures, retries, high cost, and “stuck loop” signatures.
- If you align your fields to `gen_ai.*` and `mcp.*` now, you avoid migration churn as OpenTelemetry GenAI conventions harden.
> If your agent is a distributed system, your observability has to be one too.

## What is LLM Observability?

LLM observability is the practice of instrumenting large language model-backed systems so you can understand behavior, performance, quality, and cost in production.

![A computer screen with the words back the web on it](https://cdn.sanity.io/images/vzekdneq/production/1e7ac080d91e7b43a4ad7071f6ec22fa9106c999-1200x675.webp)

For agents, it’s not “prompt + response.” That mental model is how you end up with a pile of useless logs and a compliance incident. An agent run is a sequence of decisions: planning, tool calls, retrieval, retries, policy checks, and some final outcome you can grade.

This matters more in 2026 than it did even a year ago. OpenTelemetry’s GenAI semantic conventions now live in a dedicated repository, `open-telemetry/semantic-conventions-genai`. They include explicit agent spans, events, and Model Context Protocol (MCP) context propagation guidance. That’s where the ecosystem is going. If your telemetry vocabulary drifts from it now, you’re signing yourself up for a migration later. Migration taxes are real. I hate them.

Security-wise, OWASP’s GenAI LLM Top 10 2026 dropped on **August 4, 2026**, and the project has grown to **600+ contributing experts** across **18+ countries** with **nearly 8,000** active community members. Translation: prompt/data leakage is no longer a “maybe.” It’s a thing that happens to real companies, on real timelines, with real consequences.

## Traditional vs. LLM Observability: What’s the Difference?

Traditional observability assumes requests are mostly deterministic. Same inputs, mostly same outputs. If something breaks, traces and logs usually tell a reasonably linear story.

![black and gray code padlock anchored on chain-link fence selective focus photo](https://cdn.sanity.io/images/vzekdneq/production/7f32e9504d35f355167e456a0f29d8e0e07dc404-1200x675.webp)

Agents don’t.

- **Nondeterminism is normal.** Temperature, tool timing, retrieval variance, model drift. Two “identical” runs can diverge in annoying ways.
- **Control flow is dynamic.** Agents decide what to do next. One user request can fan out into 3 tool calls or 30.
- **Payloads are radioactive.** Prompts contain user data, internal instructions, credentials you didn’t know were there, and compliance landmines.
- **Cost is a first-class metric.** Token usage and tool spend can spike silently with retries, self-corrections, or loops.
So yes, you still need traces, metrics, and logs. But you need a schema that assumes the run is messy, and defaults to not leaking sensitive data.

## The Five Pillars of LLM Observability

I like the “five pillars” framing because it forces you to stop obsessing over whether you captured the prompt, and start measuring what actually matters.

![green and silver padlock on yellow surface](https://cdn.sanity.io/images/vzekdneq/production/2d18fa42ce0ce553f583c6b7b23523f9f1eea1f4-1200x675.webp)

1. **Reliability debugging**: where runs fail, where they loop, which tool calls are flaky.
1. **Latency**: not just end-to-end. Step-level breakdowns, especially retrieval and tools.
1. **Quality**: outcomes, rubric scores, user feedback, automated checks.
1. **Cost**: tokens, retries, tool spend, per-tenant attribution.
1. **Security & compliance**: redaction, access control, retention tiers, auditability.
If you only do the first three, you’ll ship faster. If you ignore the last two, you’ll eventually have an incident. Not because you’re unlucky. Because your logging pipeline becomes your leak.

## Getting Started: the schema I’d ship in a weekend

You can implement this in two layers:

- **Application logging contract**: JSON events you emit from your agent runtime
- **OpenTelemetry mapping**: trace/span/attribute conventions that make it portable across vendors
The point isn’t “pick the perfect vendor.” The point is you have a contract your teams follow, and you can move backends without rewriting your entire debugging story.

### The schema levels

Think in four levels:

- **Run**: the whole agent invocation (one trace)
- **Step**: planning / reasoning / decide-next-action (spans)
- **Tool call**: external side effects (spans)
- **Content blobs**: raw prompt/response/tool payloads (almost never in spans)
### Agent observability logging schema table

This is the minimum viable contract. Copy it into your internal docs and make teams conform to it.

| Level | When emitted | Required fields | Optional fields | Redaction rule |
| --- | --- | --- | --- | --- |
| Run (trace root) | Start + end of agent run | `run_id`, `trace_id`, `tenant_id`, `environment`, `agent.name`, `agent.version`, `start_ts`, `end_ts`, `status` | `user_id`, `session_id`, `request_id`, `deployment.sha` | No raw user text. Store only stable IDs and hashes. |
| Step span | Each logical step | `run_id`, `span_id`, `parent_span_id`, `step.type`, `start_ts`, `end_ts`, `status` | `retry.count`, `loop.iteration`, `policy.decision` | Metadata-only by default. |
| LLM span | Each model call | `gen_ai.request.model`, `gen_ai.operation.name`, `tokens.in`, `tokens.out`, `latency_ms` | `temperature`, `top_p`, `cache.hit` | No prompts in attributes. Content via opt-in event/blob. |
| Tool span | Each tool execution | `tool.name`, `tool.status`, `latency_ms` | `tool.error_class`, `tool.cost_usd` | Allowlist tool fields. Redact everything else. |
| Retrieval span (RAG) | Each retrieval query | `retrieval.source`, `retrieval.count`, `latency_ms` | `vector_db`, `reranker`, `top_k` | Don’t log document text. Log doc IDs + hashes. |
| Content blob | Only when debugging/approval | `blob_id`, `run_id`, `blob.type`, `ttl_hours`, `kms_key_id` | `content_sha256`, `redaction.summary` | Encrypt + TTL + strict ACL. No “forever logs.” |

A few numbers to make this concrete:

- **`ttl_hours`**: I default to **24 hours** in prod for content blobs. Debuggability without building a compliance nightmare.
- **`top_k`**: log it when you do retrieval. I’ve seen RAG regressions caused by a quiet shift from **5** to **20** retrieved chunks.
- **`loop.iteration`**: if your agent can loop, you need a counter. An iteration cap of **8–12** is a sane default for many workflows.
### What NOT to log (and what to log instead)

This is where most teams screw it up.

Don’t log:

- System prompts verbatim
- Chain-of-thought / private reasoning
- Raw user content by default
- Raw tool payloads by default (especially anything that can contain credentials)
Log instead:

- `prompt_template_id` and `prompt_template_version`
- `prompt_hash` (hash of the rendered prompt after deterministic normalization)
- `input_classification` (e.g., `public`, `internal`, `restricted`)
- `tool.payload_schema_version` plus allowlisted fields
The goal is to make prompts diffable without making them readable.

## Set up tracing with OpenTelemetry (minimal but correct)

OpenTelemetry gives you a portable shape: trace → spans → events → attributes. The GenAI semantic conventions extend that shape with vocabulary that’s actually relevant to model calls and agent steps.

Authoritative baseline: the [OpenTelemetry community](https://opentelemetry.io/docs/specs/otel/) specification.

### Trace boundaries

- **One trace per agent run.**
- Root span name: `agent.run {agent.name}` (internal naming). Map to GenAI conventions inside.
### Span mapping to GenAI conventions

OpenTelemetry’s GenAI span guidance includes:

- Agent spans like plan and execute tool spans in the agent spec. See the `plan` and `execute tool` sections in the [OpenTelemetry community](https://raw.githubusercontent.com/open-telemetry/semantic-conventions-genai/main/docs/gen-ai/gen-ai-agent-spans.md).
- Client inference spans for model calls with `gen_ai.operation.name` and `gen_ai.request.model`. See [OpenTelemetry community](https://raw.githubusercontent.com/open-telemetry/semantic-conventions-genai/main/docs/gen-ai/gen-ai-spans.md).
My minimal mapping:

- **Root**: `agent.run` (INTERNAL)
- **Plan span**: `plan` (INTERNAL)
- **LLM inference span**: `{gen_ai.operation.name} {gen_ai.request.model}` (CLIENT or INTERNAL)
- **Execute tool span**: `execute_tool {tool.name}` (CLIENT if remote)
- **Retrieval span**: `retrieval {retrieval.source}` (INTERNAL)
### Attributes you should standardize

At minimum:

- Identity: `agent.name`, `agent.version`, `environment`, `tenant_id`
- Model: `gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.operation.name`
- Tokens: `tokens.in`, `tokens.out`, `tokens.total`
- Cost: `cost.usd` at span-level and `run.cost.usd` at root
- Control flow: `retry.count`, `loop.iteration`
Cost note: if you can’t attribute spend per tenant and per user, you don’t have “cost observability.” You have vibes.

### Events vs. attributes for payloads

The GenAI semantic conventions discuss strategies for capturing instructions/inputs/outputs: full buffered content, attributes, external storage, or streaming chunks.

The boring answer is the right one: **metadata in attributes; content in external blobs**.

Use span attributes for:

- hashes
- lengths
- MIME types
- schema versions
Use events only when:

- you’re in a controlled dev environment
- you have strict access controls
- you need just enough content for a short time
And even then, redact first.

## How do you trace tool calls in an LLM agent?

Tool calls are where your agent stops being “chat” and starts being “software.” They’re also where you leak secrets if you log like it’s 2015.

A practical approach:

1. Create a tool span for every tool invocation.
1. Put timing, status, and a stable request ID on the span.
1. For request/response payloads, log only allowlisted fields (think: `resource_id`, `query_type`, `result_count`).
1. Store full payloads as encrypted blobs with TTL, and only when explicitly enabled.
If you’re using MCP tools/servers, OpenTelemetry recommends context propagation by injecting `traceparent`, `tracestate`, and `baggage` into `params._meta` and extracting it on the receiver as the remote parent. See the MCP conventions doc by the [OpenTelemetry community](https://raw.githubusercontent.com/open-telemetry/semantic-conventions-genai/main/docs/gen-ai/mcp.md).

That one detail is what makes your tool spans line up cleanly across process boundaries.

## How do I avoid logging secrets and PII in LLM prompts and tool outputs?

Treat telemetry as a data exfil path. Because it is.

Here’s the defense-in-depth model that actually holds up under reality (meaning: under deadlines, new services, and humans making mistakes).

### 1) Allowlist-first logging

For each tool, define:

- `tool.payload.allowlist.paths` (JSONPaths you permit)
- `tool.payload.denylist.paths` (high-risk fields you always drop)
Assume “log nothing” unless explicitly allowed.

### 2) Redaction transforms (SDK-side)

Before anything leaves the process:

- redact credentials (`Authorization`, `api_key`, `token`, `cookie`)
- redact common PII patterns (emails, phone numbers)
- truncate long strings (cap at **256** chars for any field)
- hash stable identifiers (SHA-256) when you need correlation without disclosure
### 3) Redaction transforms again (collector-side)

Someone will bypass your SDK. Or a new service will ship without it. So you do it again in the OpenTelemetry Collector:

- drop known sensitive attributes
- enforce max attribute size
- enforce environment-based policies (prod stricter than dev)
### 4) Don’t put secrets in prompts

This sounds obvious until you see real systems.

Secrets belong in a secret manager and should never be embedded into prompts. The authoritative baseline is [Google Cloud](https://cloud.google.com/secret-manager/docs/best-practices): minimize access, rotate, audit, and never leak via logs.

## What are spans and traces in OpenTelemetry?

- A **trace** is a tree representing one end-to-end operation.
- A **span** is one timed unit of work within that trace.
For agents:

- trace = one run
- spans = planning step, model call, retrieval, tool execution, policy check
- logs/events = details you might want to attach (carefully)
If you adopt that mental model, most schema decisions stop being mysterious.

## What is tail-based sampling and when should I use it?

Head-based sampling decides at the start. Tail-based sampling decides after the trace finishes, based on what happened.

For agents, tail sampling is ridiculously effective because “interesting” traces are rare but expensive:

- failures
- high retry count (>= **2**)
- loop iterations above a threshold (>= **6**)
- high cost (e.g., `run.cost.usd` > **$0.25** for consumer workflows)
- policy violations
Default: head sample low (say **1–5%**) for volume control.

Then: tail sample 100% of “bad” runs so you can debug what matters without setting money on fire.

## How do you correlate logs with traces for debugging?

Use three IDs everywhere:

- `trace_id` (OTel)
- `run_id` (your business identifier)
- `request_id` (edge ingress)
Logs should include `trace_id` and `span_id`. Your agent events should include `run_id`. This lets you start from a user complaint, find the run, then pivot into traces.

And yes, this is why “one trace per run” is such a powerful simplification.

## Trace a RAG application (agent + retrieval)

RAG (Retrieval-Augmented Generation) is where observability gets subtle.

You need to see:

- retrieval latency
- retrieval count (`top_k`)
- which index/collection you hit
- reranker model (if any)
- how many tokens you spent stuffing context
But you do **not** want raw retrieved chunks in traces.

Instead:

- log `doc_id` and `doc_hash`
- log `chunk_id` and `chunk_hash`
- log `retrieval.query_hash`
Then store actual content only behind explicit gates.

If you want a deeper security playbook here, I’ve already written the adjacent piece on [RAG](/blog/prevent-sensitive-data-leakage-rag).

## View traces (what “good” looks like)

When you open a trace view for a run, you should be able to answer in under **60 seconds**:

- Did it succeed?
- Where did time go?
- Where did cost go?
- Which tool failed?
- Did it loop?
- Was a policy gate triggered?
If you can’t, you’re collecting trivia, not observability.

For deeper control-flow patterns (retries, checkpoints, HITL), see [AI agents](/pillars/ai-agents) and my guide on [AI in production](/pillars/ai-engineering-production).

## Monitor performance (latency + cost)

Monitoring agents is mostly about catching regressions before your users do:

- P95 step latency changes
- token usage drift
- tool error rate spikes
- loop frequency
I keep two dashboards:

1. **Run-level SLOs**: success rate, p95 end-to-end latency, p95 cost.
1. **Step-level heatmap**: which spans are dominating time/cost.
If you want a measurement discipline for LLM latency, I wrote up the methodology I use in [LLM latency](/blog/llm-latency-benchmark-methodology).

For cost math (including retries and tools), connect this with LLM cost and the practical breakdown in AI in production.

## Default-on vs opt-in logging in production for agents

Here’s my stance: **production should be metadata-only by default.** If you need content, you earn it with explicit controls.

Default-on (prod):

- spans with timings + statuses
- token counts
- cost estimates
- tool names + status codes
- prompt/template IDs + hashes
Opt-in (prod, time-limited):

- full prompts/responses
- full tool payloads
- retrieved content
And when you enable opt-in, it should be:

- scoped to a tenant/user/session
- time-boxed (e.g., **1 hour** window)
- stored as encrypted blobs with TTL
## How to store full prompt/response safely (without bloating traces)

The pattern that scales:

- Put raw content in object storage (S3/R2/GCS).
- Make it content-addressed (`sha256` as key).
- Encrypt with KMS.
- Apply TTL lifecycle policy.
- Store only `blob_id` plus `content_sha256` plus `ttl_hours` in your traces.
This keeps traces lightweight and searchable, and keeps sensitive data behind a stricter access model than “anyone with read access to logs.”

## Retries, self-corrections, and agent loops: how to represent them

A few rules:

- A single logical model call with retries should be one inference span, with `retry.count` and retry events.
- Agent loops should increment `loop.iteration` and emit a loop span per iteration.
- If the agent is stuck (same tool call repeated), emit a `stuck.signature_hash` so you can alert on it.
Once you do this, tail sampling becomes mechanical. “Keep any trace where `loop.iteration >= 6`.” Easy.

## How to design a redaction pipeline (SDK + collector)

This is the part I want teams to stop hand-waving.

1. **SDK-side**: redact early so secrets never leave process memory.
1. **Collector-side**: redact again because someone will ship without the SDK.
1. **Backend-side**: enforce access control and retention policies.
I’ve built the 7-agent blog publishing pipeline for this site, and one of the biggest lessons was boring: deterministic gates catch issues earlier than “we’ll review it later.” Same idea here. Put deterministic redaction and drop rules in front of your exporter, not as a best-effort afterthought.

To go deeper on agent security failure modes, connect this to my broader [AI security](/blog/ai-security-complete-guide) and the specific risk of [prompt injection](/blog/prompt-injection-2026-owasp-llm-vulnerability).

### Internal links you’ll likely want next

- [AI agents](/pillars/ai-agents)
- [AI in production](/pillars/ai-engineering-production)
- [agentic AI](/blog/rise-of-agentic-ai)
- [agent orchestration](/blog/multi-agent-ai-systems-production)
- [OpenTelemetry](/blog/opentelemetry-ai-agents-instrumentation)
- [RAG](/blog/prevent-sensitive-data-leakage-rag)
- [prompt injection](/blog/advanced-prompt-injection-techniques-2026)
- [Claude Code](/blog/claude-code-security-2026)
- [local LLM](/blog/local-llms-complete-guide)
Photo by Luca Bravo on Unsplash.

## FAQ

### What is agent observability?

Agent observability is the ability to understand what an AI agent did during a run: which steps it took, which tools it called, how long each step took, and why it succeeded or failed. It’s broader than logging prompts and responses because agents have dynamic control flow and external side effects.

### What should I log for an AI agent run?

Log run metadata (IDs, environment, agent version), step timings (spans), tool call status, token usage, and cost attribution. Avoid logging raw prompts and raw tool payloads by default. Use hashes, template IDs, and short-lived encrypted blobs when you need content.

### How do you trace tool calls in an LLM agent?

Create one span per tool invocation and attach a stable request ID, latency, and status. Log only allowlisted fields from tool inputs/outputs. If tools run out-of-process (especially over MCP), propagate trace context so tool spans connect back to the agent trace.

### How do I use OpenTelemetry for LLM/agent tracing?

Use one trace per agent run and spans for planning, model inference, retrieval, and tool execution. Adopt `gen_ai.*` attributes for model calls and align span naming with the GenAI semantic conventions. Keep content out of span attributes and store it externally when needed.

### How do I avoid logging secrets and PII in LLM prompts and tool outputs?

Use allowlist-first logging, redact in the SDK before exporting, and redact again in the collector as a backstop. Don’t embed secrets in prompts in the first place. Store any necessary raw content as encrypted blobs with strict access control and a short TTL.

### What is tail-based sampling and when should I use it?

Tail-based sampling keeps or drops traces after they finish, based on what happened inside them. It’s useful for agents because you can keep 100% of failed, high-cost, or looping runs while sampling most routine runs at a low rate. This preserves debuggability without exploding telemetry volume.