# Agent Readable Documentation Toolchain [2026]: My Stack

> A docs-as-code pipeline that stays great for humans, useful for your trusted AI agents, and aggressively unhelpful to random scrapers. Practical, not theoretical.

- Canonical: https://www.kunalganglani.com/blog/agent-readable-documentation-toolchain
- Author: Kunal Ganglani
- Published: 2026-09-08 · Updated: 2026-09-08
- Category: Developer Tools · Tags: documentation, knowledge-base, ai-agents, anti-scraping, developer-tools

## TL;DR

An agent readable documentation toolchain is a way to publish documentation so it works for people and for trusted AI assistants at the same time. The trick is to ship clean, intentional artifacts (Markdown, diagrams-as-code, a simple index, and an llms.txt file) instead of letting bots scrape messy HTML. Pair that with real access control, rate limits, and bot filtering so random crawlers don’t overload your site or steal your content. Then add tests in CI so docs don’t drift and teach your agents the wrong thing. The result is docs that stay pleasant to read, easier to search, and safer to automate against.

## Agent Readable Documentation Toolchain [2026]: My Stack

Agent readable documentation toolchain is a docs pipeline that produces **two first-class outputs**: a human-pleasant documentation site and a machine-pleasant, verifiable bundle that trusted agents can consume. If you don’t design for both, you’ll either ship unreadable “AI docs” or you’ll wake up to 100k bot requests/day and a broken site.

![What is an agent readable documentation toolchain? — section illustration](https://cdn.sanity.io/images/vzekdneq/production/4884cd09d61e86a0a3a1c92b8ece4f0c96f88e32-1200x675.webp)

**Key takeaways**

- Agent-readable docs are not “HTML that an LLM can kinda parse”. They are explicit artifacts: Markdown, manifests, stable IDs, and predictable retrieval paths.
- Robots.txt is polite fiction. If you care about scraping, you need **layered controls** at the edge: auth, rate limits, WAF rules, and bot challenges.
- Diagram-as-code only becomes agent-friendly when you ship the **source + rendered output + searchable text** together.
- Run lexical search (BM25) and semantic search (embeddings) side-by-side. They fail differently, which is exactly what you want.
- Treat docs like production code. If you don’t test them in CI, your agents will learn the wrong thing faster than your humans will notice.
## What is an agent readable documentation toolchain?

An **agent readable documentation toolchain** is a docs-as-code pipeline that turns your source docs into a set of artifacts optimized for software agents: clean Markdown, structured metadata, diagram sources, search indexes, and a controlled “agent entry point” like `llms.txt` and a JSON manifest.

![Why this matters in 2026: “helpful to our AI, hostile to everyone else’s bots” — section illustration](https://cdn.sanity.io/images/vzekdneq/production/4884cd09d61e86a0a3a1c92b8ece4f0c96f88e32-1200x675.webp)

Here’s the checklist version I recommend (this is the part most “docs-as-code” posts skip):

1. **Authoring:** Markdown with strict frontmatter, stable headings, and canonical URLs.
1. **Diagrams-as-code:** Mermaid or PlantUML in-repo, compiled in CI to SVG/PNG.
1. **Human site build:** Static rendering (Astro/Docusaurus/MkDocs) with good navigation.
1. **Agent bundle build:** Markdown export + manifest JSON + `llms.txt` + diagram sources.
1. **Search:** Local BM25 for exactness and embeddings for fuzziness.
1. **Controls:** Tiered public/private docs + edge auth + rate limits + bot filtering.
1. **Integrity:** Link checks, snippet tests, diagram compile tests, drift detection.
I’m opinionated about one thing: **agents should read artifacts you intentionally publish**, not scrape whatever HTML happens to be on your site today.

> If you want agents to be reliable, stop letting them “interpret” your docs. Give them artifacts that are meant to be parsed.

## Why this matters in 2026: “helpful to our AI, hostile to everyone else’s bots”

The pressure on docs teams is coming from both sides.

![Markdown structure that agents can cite (without ruining human UX) — section illustration](https://cdn.sanity.io/images/vzekdneq/production/4884cd09d61e86a0a3a1c92b8ece4f0c96f88e32-1200x675.webp)

On one side, internal tooling wants your documentation to be machine-consumable for **AI agents**, internal copilots, and RAG systems. On the other side, the open web is being strip-mined by indiscriminate scrapers. The result is a perverse outcome: teams either lock everything down (hurting humans) or open everything up (feeding bot farms).

The SERP for the exact phrase is sparse, but the demand is real. Per our Search Console neighborhood analysis, kunalganglani.com already has **~94 related impressions** and a **best related average position of ~7.1**, with an estimated **~290 searches/month** across adjacent queries. That’s not “mass market” traffic. It’s the exact kind of high-intent, engineer-driven query I like.

There’s also a distribution risk that’s getting louder. Hacker News has been full of posts about independent wikis getting effectively buried and rate-limited by platforms. If you rely on “Google will route people to our docs,” you’re betting your onboarding and support costs on an external ranking algorithm.

## Markdown structure that agents can cite (without ruining human UX)

Markdown is still the boring answer. It’s also the right answer.

What makes Markdown **agent-readable** isn’t the fact that it’s plain text. It’s that you can impose structure without fighting your renderer. The rules I enforce:

- **Frontmatter is non-negotiable.** Every page gets `title`, `description`, `canonical`, `updated`, and an access tier like `visibility: public|internal|restricted`.
- **Stable heading IDs.** Your renderer should produce deterministic anchor links. If your site auto-generates random slugs or changes them across builds, agents can’t deep-link reliably.
- **Canonical URLs everywhere.** If a page exists in multiple places (marketing site, docs, internal wiki), decide which is canonical and bake it in.
- **One concept per page.** Agents chunk anyway. You’re better off writing in chunks you’d want retrieved.
I’ve shipped enough “knowledge base” systems to know that most pain isn’t writing. It’s **drift** and **findability**. You can paper over that for humans with navigation UX. Agents don’t get that luxury.

Practical number: I try to keep “retrieval-sized” sections in the **150–400 word** range, separated by headings that would make sense as standalone citations.

If you’re already thinking about [RAG](/glossary/rag) and retrieval-augmented generation for internal copilots, this is the same discipline. Your docs pages are the corpus.

## Diagram-as-code that agents can actually use

Diagram-as-code fails when the rendered diagram is the only thing that survives.

A human can look at an SVG and understand it. An agent needs the **graph structure** or at least the source text. So the pipeline has to treat diagrams like build artifacts:

- Keep Mermaid (`.mmd`) and PlantUML (`.puml`) source files in-repo.
- Compile them in CI to **SVG** (preferred for web) plus a raster fallback if you need it.
- Store the compiled outputs adjacent to the source, so diffs and blame stay obvious.
- Generate an **agent-friendly text summary** per diagram (even a short paragraph) and index that summary.
For Mermaid, `mermaid-cli` is the most common hammer. The repo is active and widely used (Mermaid CLI). For PlantUML, the CLI workflow is similarly straightforward from the official project ([Arnaud Roques](https://plantuml.com/), PlantUML creator).

Here’s the part people miss: agents don’t just need “a diagram exists”. They need to answer questions like:

- “Which service calls `billing-api`?”
- “Where is auth enforced?”
- “Is this async or sync?”
If that information only lives in pixels, you’ve built diagram-as-code for git diffs, not for AI agents.

## llms.txt: the agent entry point that’s becoming table stakes

`llms.txt` is a proposed standard file placed at `/llms.txt` that provides a curated, LLM-friendly directory of your site. The spec is authored by [Jeremy Howard](https://llmstxt.org/) and has evolved into v2 with adoption patterns and linking conventions (`rel="alternate" type="text/markdown"`, `rel="describedby"`).

My stance: **publish `llms.txt` even if you block most bots**.

Why? Because it gives you a narrow, auditable choke point.

- Humans browse the full HTML site.
- Trusted agents fetch `llms.txt` and then pull specific Markdown endpoints.
- Untrusted scrapers hit the HTML and get rate-limited, challenged, or blocked.
You’re not “hiding” information. You’re making the machine path explicit and controllable.

This also aligns with what I keep seeing in agent reliability research. As [Dan Luu](https://danluu.com/agentic-testing/) shows, agents don’t magically become correct just because you tell them to use better techniques. They need verification, guardrails, and high-signal inputs. Clean docs artifacts are part of that.

## Local search + semantic search, side-by-side

Search is where the toolchain becomes real.

If you only do embeddings, you’ll get “feels right” retrieval that fails silently. If you only do lexical search, you’ll miss synonyms, abbreviations, and messy human queries.

So I run them together:

- **BM25** (lexical): fast, great for exact API names, error codes, config keys.
- **Embeddings** (semantic): better for “what’s the difference between X and Y?” questions.
Hybrid retrieval isn’t fancy anymore. It’s baseline for [production AI](/pillars/ai-engineering-production) because it gives you two independent failure modes.

Concrete example: if someone searches “webhook retries ordering”, BM25 will nail it because those words exist verbatim. Semantic search might over-generalize to generic “reliability” docs. Conversely, “why do we use idempotency keys” often benefits from semantic retrieval.

If you want a deeper architectural take on this, my [LLM knowledge base architecture guide](/blog/llm-knowledge-base-architecture-guide) lays out why wiki vs notes vs RAG is mostly a retrieval and governance decision.

## Anti-scraping beyond robots.txt: layered, boring controls

Robots.txt is a request. Scrapers don’t have to care.

If you want to **block AI scrapers documentation** style, you need layers. My default stack for a static docs site:

1. **Tiered paths:** `/docs/public/` vs `/docs/internal/`.
1. **Edge auth for private docs:** Cloudflare Access is a clean option for SSO-gated docs without building auth yourself. Cloudflare even dogfoods agent-facing patterns in their own docs and publishes `llms.txt` (see the Cloudflare docs index and `/llms.txt` reference in their docs: [Cloudflare One docs](https://developers.cloudflare.com/cloudflare-one/identity/)).
1. **Rate limiting:** per-IP and per-path. Give `/llms.txt` and `/manifest.json` stricter budgets.
1. **WAF bot rules:** challenge obvious headless clients, bad ASNs, weird request patterns.
1. **Token-gated agent endpoints:** downloadable bundles require a short-lived token.
If you want a hands-on implementation angle, I wrote a full guide on running a WASM bot filter: [How to run Anubis WASM bot filter](/blog/anubis-wasm-bot-filter-reverse-proxy). That’s the kind of tooling that actually moves the needle beyond “please don’t crawl me”.

Ethics note: I’m not advocating for dark patterns that hurt humans. The goal is “humans get fast docs, trusted agents get clean artifacts, random bots get nothing useful.” That’s defensible.

## Public docs + private knowledge: stop pretending it’s one thing

Most teams have at least three documentation classes:

- **Public:** API docs, SDK usage, onboarding, pricing-related behavior.
- **Internal:** runbooks, incident notes, architecture tradeoffs.
- **Restricted:** credentials processes, security controls, customer-specific details.
Mixing these in one site and praying your ACLs are perfect is how you end up with leaks.

I’ve learned (the hard way) that compliance is easier when it’s baked into the scaffolding. At Rise People, I built a SOC 2-compliant project scaffolding CLI that was adopted org-wide. The big lesson: **controls implemented at creation time beat reviews at PR time**.

Apply that to docs. Make visibility a required field. Make “public by default” impossible for certain folders. Enforce it in CI.

And if you’re building agent workflows around this, treat it as AI security and LLM security, not “just docs.”

## Docs CI: tests that prevent drift (and keep agents honest)

Docs rot faster in an agent-first world because agents amplify stale information.

So I treat docs like code:

- **Link checking** (internal and external) on every PR.
- **Snippet tests** for code blocks that claim “run this”.
- **Diagram compile tests** for Mermaid/PlantUML so broken diagrams don’t ship.
- **API example validation** against an OpenAPI schema when possible.
- **Drift detection**: if an API endpoint changes, the docs PR should fail until docs are updated.
This is directly aligned with Dan Luu’s broader point: you don’t get reliability by vibes. You get it by verification ([Dan Luu](https://danluu.com/agentic-testing/)).

If you’re already doing [AI in production](/pillars/ai-engineering-production), this is the same mindset as eval gates. I wrote about it more generally here: [AI engineering evals: regression gates](/blog/ai-engineering-evals-gates).

## A safe agent endpoint: bundles, manifests, and MCP

Once you’ve got clean Markdown and diagrams, you need a distribution format that doesn’t force agents to crawl HTML.

The model I like is **three agent-facing artifacts**:

- `/llms.txt` (public or semi-public): the curated directory.
- `/agent/manifest.json` (usually gated): a machine-readable index of pages, their canonical URLs, hashes, and visibility.
- `/agent/bundle.tar.gz` (gated): a downloadable snapshot of the Markdown corpus and diagram sources.
Make the manifest include at least:

- `id`
- `canonical_url`
- `markdown_url`
- `updated_at`
- `sha256`
- `visibility`
- `parents` (for hierarchy)
That `sha256` matters. It gives you integrity checks and makes it easy to do incremental updates.

If you want interactive access instead, you can expose a tool interface via an MCP server. Do it carefully. It’s an authenticated API surface, which means it deserves a real threat model. Start with my broader [AI agent attack surface checklist](/blog/ai-agent-attack-surface-checklist) and the deeper [agent-specific attack surfaces](/blog/agent-attack-surfaces-security).

One more experience-backed note: running this blog’s tooling ecosystem taught me that small, explicit artifacts compound. I maintain an LLM pricing tracker at [kunalganglani.com/llm-prices](/llm-prices) and 25+ free utilities under [/tools](/tools). The surprising lesson is that **utility + structure beats volume**. Docs should steal that playbook. Give both humans and agents something they can rely on.

## My reference architecture (the pipeline that survives AI scrapers)

Here’s the end-to-end view:

- **Repo**: `/docs` markdown + `/diagrams` (Mermaid/PlantUML)
- **Build step 1**: lint markdown + enforce frontmatter schema
- **Build step 2**: compile diagrams to SVG + generate per-diagram text summaries
- **Build step 3**: build static site (human UX)
- **Build step 4**: generate agent bundle + `llms.txt` + manifest + checksums
- **Build step 5**: build BM25 index (public site search) + embeddings index (internal RAG)
- **Deploy**:
  - Public docs on CDN
  - Private docs behind Cloudflare Access / basic auth
  - Agent endpoints token-gated + rate-limited
  - Bot mitigation at the edge
If you want to connect this to broader agent architecture, start with [AI agents](/pillars/ai-agents), then agent orchestration, then my concrete debugging pattern: [execution trace tree for AI agents](/blog/execution-trace-tree-agents).

My prediction: within 12–18 months, “docs-as-code” without an explicit agent artifact path will feel as outdated as shipping a REST API without OpenAPI. The teams that win will be the ones who keep their docs delightful for humans while making them **intentionally consumable** for trusted machines.

Photo by James Harrison on Unsplash.

## FAQ

### What is an agent readable documentation toolchain?

It’s a docs pipeline that produces machine-friendly artifacts on purpose, not by accident. You publish clean Markdown, stable links, indexes/manifests, and controlled endpoints so trusted agents can retrieve and cite docs reliably.

### How do you make documentation agent-readable without ruining human UX?

Keep the human site as the primary experience, then generate an agent bundle in parallel. Humans get navigation and design. Agents get Markdown, manifests, and predictable URLs like `/llms.txt` and `.md` alternates.

### What is llms.txt and where does it fit in a docs pipeline?

`llms.txt` is a proposed standard file at the site root that gives agents a curated directory and guidance for consuming your content. It’s best treated as the official entry point for trusted agent access, alongside a manifest or downloadable bundle.

### What are practical layers of anti-scraping beyond robots.txt?

Use edge authentication for private docs, rate limits per path, WAF rules for bot patterns, and token-gated endpoints for agent bundles. Robots.txt can still exist, but it’s only one weak layer in a real defense.

### How do you run local search (BM25) and semantic search (embeddings) side-by-side?

Use BM25 for exact terms like API names and error codes, and embeddings for fuzzier queries and conceptual questions. Query both and merge results, or fall back from one to the other when confidence is low.

### How do you test docs in CI to prevent drift?

Run link checks, validate frontmatter, compile diagrams, and execute code snippets where possible. Add drift checks that fail builds when APIs change but docs don’t, because agents amplify stale guidance quickly.