# How to Build a Browser Based AI Agent (Safe-by-Default) [2026]

> A practical browser AI agent architecture that stays local: Worker planner, Wasm sandbox, IndexedDB memory, and a service worker tool bridge with real guardrails.

- Canonical: https://www.kunalganglani.com/blog/build-browser-ai-agent
- Author: Kunal Ganglani
- Published: 2026-09-25 · Updated: 2026-09-25
- Category: Frontend and Mobile · Tags: ai-agents, mcp, webassembly, service-worker, indexeddb

## TL;DR

A browser-based AI agent runs in your web app instead of your servers, so it can feel instant, work offline, and keep user data local. The hard part isn’t the model. It’s safety. If the agent can call tools (network, files, accounts), prompt injection or XSS can turn into real actions. The practical fix is an architecture with strong boundaries: put planning in a Web Worker, store memory in IndexedDB with eviction in mind, and route every tool call through a service worker that enforces allowlists, validation, and human approvals for risky actions.

If you want **how to build a browser based AI agent** that feels instant, works offline, and doesn’t send user data back to your servers, you can. But if you build it like a typical “agent tutorial” (planner on the main thread, tools called with `fetch()` wherever), you’re basically shipping an XSS-shaped foot-gun with a chatbot UI.

The prerequisite that trips people up is not “pick the right model.” It’s this: **you need cross-thread boundaries** (Web Worker + Service Worker) or your “agent” ends up with ambient authority over everything.

This is the safe-by-default reference architecture I wish existed. It keeps the agent local to the browser, persists memory without pretending storage is durable, and makes tool calls auditable, confirmable, and boring.

### How to build a browser based AI agent (safe-by-default)

1. **Choose your model execution mode**: local (WebGPU/Wasm) for privacy/offline, or remote API for quality. Support both.
1. Put the **planner** (LLM loop + tool selection) in a **Dedicated Web Worker**.
1. Store memory in **IndexedDB** with explicit compaction rules. Assume eviction.
1. Request **persistent storage** only when you have critical local state to protect.
1. Implement tools behind a **Service Worker tool bridge**. No direct `fetch()` from the planner.
1. Enforce **allowlists + JSON Schema validation** for every tool call.
1. Add **human-in-the-loop confirmations** for consequential actions (money, sharing, deletion, account changes).
1. Sandbox any “user code” or third‑party tool bundles with **WebAssembly or WebContainers**.
1. If you need enterprise data/tools, bridge to **Model Context Protocol (MCP)** through an HTTPS/WebSocket relay.
1. Log an **audit trail locally** (redacted) so the user can see what happened.
## What is a browser based AI agent?

A **browser based AI agent** is an agentic app where the inference or tool-planning loop runs on the client, uses browser storage for memory, and calls tools through web-native capabilities (APIs, service workers, extensions) instead of a server-side orchestrator.

![logo](https://cdn.sanity.io/images/vzekdneq/production/a12327118014e41802e55640141e3323fb52a421-1200x675.webp)

I’m bullish on this pattern for two boring reasons: cost and privacy.

Server-side agents become an **LLM cost** treadmill the second you get real traffic. Browser-native agents push compute and data to the edge where it belongs.

The catch is also boring: the browser is hostile by default. Your agent reads untrusted text all day. If you don’t build hard boundaries, prompt injection turns into unintended actions. That’s not “AI safety.” That’s just… software safety.

## Tool calling basics: the model requests, your app executes

Most agent tutorials accidentally imply the model runs code. It doesn’t.

![the best way to build web apps without code](https://cdn.sanity.io/images/vzekdneq/production/e23c18a64106ad26bdc7309b38445d1046d12482-1200x675.webp)

As **Rohini Gaonkar** (writing for AWS) puts it: _the model is the decision-maker and your code is the hands_. The model emits a structured tool request. Your runtime decides whether to execute it, validates parameters, applies policy, runs the tool, and returns the result back to the model.

That separation matters more in the browser than anywhere else because:

- The “hands” run with ambient authority (cookies, user session, local storage).
- The model can be steered by untrusted web content.
- Your UI is one click away from real damage.
### The agent loop you actually need in the browser

A browser agent loop is still the classic 4-step cycle, just split across threads:

1. **Observe**: user input + tool results + selected memory items.
1. **Plan**: model produces either a message or a tool call.
1. **Act**: runtime executes tool (or asks for confirmation).
1. **Reflect**: store a summary, update embeddings, update audit log.
One production detail people gloss over: retries.

When I spearheaded an AI platform for short video generation and live-stream commerce, the bill was dominated by **retries and regeneration**, not first-pass tokens. In the browser, retries dominate something more important than cost: user trust. A tool that fails twice feels haunted, even if your error rate is “technically fine.”

## Browser AI agent architecture (what runs where)

Here’s the reference layout I recommend for a **browser AI agent architecture** that doesn’t collapse into spaghetti:

![the best way to build web apps without code](https://cdn.sanity.io/images/vzekdneq/production/e23c18a64106ad26bdc7309b38445d1046d12482-1200x675.webp)

- **UI thread (main window):** chat UI, permission prompts, transaction previews.
- **Planner Worker (Dedicated Web Worker):** prompt assembly, model calls, tool selection, loop control.
- **Tool Bridge (Service Worker):** the only place allowed to do network fetches for tools. Enforces policy.
- **Memory (IndexedDB + Cache API):** episodic logs, semantic memory, tool call audit trail.
- **Sandbox (Wasm/WebContainers/iframe):** optional isolation for untrusted code/tools.
- **Model runtime:** local WebGPU/Wasm (preferred for privacy/offline) or remote API.
This separation does two things I care about:

1. Keeps the UI responsive. Heavy inference or parsing never touches the main thread.
1. Shrinks the blast radius. If your UI gets XSS’d, it shouldn’t automatically become “full agent root.”
MDN describes service workers as proxy servers that sit between the app, the browser, and the network, and can intercept requests. That’s exactly what we want. A policy-enforcing choke point. See the [MDN contributors](https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API) docs for the canonical definition.

[Image: Architecture diagram — UI ↔ Worker planner ↔ Service worker tool bridge ↔ Tools + IndexedDB]

## In-browser LLM: local inference vs remote API (and why hybrid wins)

Yes, you can run an LLM completely in the browser. Projects like WebLLM do real in-browser inference using WebGPU/Wasm.

The trade is simple:

- **Local inference:** best privacy, offline-capable, zero server spend. Slower, tighter memory limits.
- **Remote API:** best quality and speed, but you pay per token and you leak data unless you’re disciplined.
My opinionated take: build hybrid from day one.

- Default to **remote** for “power mode” tasks.
- Offer **local** as privacy mode and offline fallback.
If you’ve been following my [local LLM](/pillars/llm-hardware-local-ai) work, you already know why I’m biased here. Based on the benchmark data I maintain at **https://www.kunalganglani.com/llm-benchmarks**, the gap between “toy local” and “usable local” is mostly about latency and memory, not model intelligence. In-browser makes those constraints painfully obvious.

Practical numbers to plan around:

- Plan for **2–8 GB** of model assets for a small-ish usable model with caches, depending on quantization.
- Keep your first response under **500 ms** perceived latency if you want it to feel instant.
- Target **60 fps** UI. If inference steals the main thread for **16 ms** chunks, it will feel broken.
### How do you run heavy AI inference without blocking the UI?

Three rules:

1. **Never** run inference on the main thread.
1. Put inference in a Worker. If you need shared memory, you’re in **SharedArrayBuffer** land.
1. Keep the planner separate from rendering. Stream tokens to the UI via `postMessage`.
## IndexedDB agent memory: schema, compaction, quotas, persistence

“Just store it in IndexedDB” is advice from people who have never been paged for data loss.

Browsers enforce quotas and evict origin data under storage pressure. MDN’s storage quota and eviction guide is blunt about it. Don’t assume your data is permanent. See [MDN contributors](https://developer.mozilla.org/en-US/docs/Web/API/Storage_API/Storage_quotas_and_eviction_criteria).

### Is IndexedDB persistent? How much can it store?

IndexedDB is _durable-ish_, not durable. In practice:

- Quotas vary by browser and device.
- Eviction can happen when disk is low.
- Users manually clearing site data is still the most common “delete.”
Design like you have **0 bytes guaranteed** and you’ll build the right system.

### A memory schema that survives real usage

Split memory into four stores:

1. **conversations**: raw turns (bounded). Keep last **N=200** turns, then summarize.
1. **summaries**: rolling summaries per thread/day/week. Store at **~1–4 KB** each.
1. **semantic_memory**: embedding vectors + metadata + source pointers.
1. **tool_audit**: immutable tool call log with redaction.
Compaction policy that works in practice:

- Summarize every **20** user turns.
- Delete raw turns older than **7 days** unless pinned by user.
- Keep embeddings for pinned items only.
- Keep tool audit for **30 days** by default.
If you want deeper memory patterns, I wrote more about agent state in [AI agents](/pillars/ai-agents) and [RAG](/glossary/rag).

### Requesting persistent storage (when you should, and when you shouldn’t)

If your agent stores anything that would genuinely hurt to lose (encrypted notes, offline tasks), request persistent storage.

- Call `navigator.storage.persist()` only after the user opts into offline mode.
- Explain what you’re storing and why.
web.dev explains that browsers may remove data from IndexedDB/Cache under pressure, and persistent storage reduces that eviction risk. See the guidance from [Pete LePage](https://web.dev/persistent-storage/).

[Image: Memory compaction flow — raw turns → summary → embeddings → eviction-aware retention]

## The service worker tool bridge (with allowlists, confirmations, audit logs)

If you take one thing from this post: **do not let the model planner call network tools directly**.

Put all tool I/O behind the service worker. The planner Worker sends a tool request over `MessageChannel`. The service worker validates, enforces policy, and executes.

That one constraint cleans up a ton of problems: security, debuggability, even product UX. You finally have one place to put “no, you can’t do that” logic.

### Safest way to expose tools in the browser

A tool should be defined by:

- A **name** (`calendar.create_event`)
- A **schema** for parameters (JSON Schema)
- An **egress policy** (allowed hosts, methods)
- A **risk level** (`low`, `medium`, `high`)
Minimum set of guardrails:

- **Allowlist** domains. Default deny.
- **Method allowlist** (`GET` only unless explicitly allowed).
- **Parameter validation**. Reject unknown fields.
- **Timeouts**. Hard cap at **10 seconds**.
- **Rate limit** per tool: e.g. **30 calls/min**.
### Human-in-the-loop confirmations for consequential actions

Consequential actions need a friction bump. Not a sad “are you sure?” modal. A real preview.

Patterns that actually work:

- **Transaction preview**: show the exact request (URL, method, body) before send.
- **Scoped approval**: “Allow this tool for 10 minutes” vs forever.
- **Two-step commit**: tool returns a “draft,” user confirms commit.
In my Walmart conversational commerce chatbot work, the biggest quality wins came from retrieval quality, not model choice. The security corollary is similar. **Policy quality, not model choice, dominates safety** once you have real users hammering your edge cases.

### Local audit logging without leaking secrets

Log every tool call, but redact aggressively:

- Store request metadata (tool name, host, status, duration in **ms**).
- Store a hashed payload (SHA-256) instead of raw body.
- Store the user confirmation decision (`approved`/`denied`).
If you’re serious about this, align with the patterns in my [AI in production](/pillars/ai-engineering-production) content and the logging schema ideas in [AI agent observability logging schema](/blog/ai-agent-observability-logging-schema).

## MCP bridge: connecting browser agents to real systems safely

Model Context Protocol (MCP) is a good idea that gets misused.

MCP is an open standard for connecting AI assistants to the systems where data lives, including content repositories, business tools, and development environments. That’s from the original announcement by [Anthropic](https://www.anthropic.com/news/model-context-protocol).

Browser constraint: no raw TCP. Your agent can’t just open a socket to a local MCP server.

### The relay pattern that doesn’t terrify your security team

Use an MCP relay:

- Browser talks to your relay over **HTTPS/WebSocket**.
- Relay talks to MCP servers inside a controlled network.
- Relay issues **origin-bound tokens** and enforces per-tool ACLs.
Hard rules:

- Never expose “power tools” by default. Capability-based exposure only.
- Bind tokens to origin + user + tool scopes.
- Keep tool schemas in the browser so the planner can’t invent parameters.
If you’re building MCP seriously, you’ll want my [MCP server security best practices](/blog/mcp-server-security-best-practices) and [How to secure MCP servers: Auth + AuthZ](/blog/mcp-server-authentication-authorization).

Here’s a good walkthrough:

[Watch: How Model Context Protocol (MCP) actually works](https://www.youtube.com/watch?v=cGuyrANVi4A)

## Sandboxing: Web Workers vs iframes vs WebAssembly vs WebContainers

At some point you’ll want “tools” that are more than HTTP calls. User-provided scripts. A tiny rules engine. A plugin ecosystem that third parties can extend.

That’s where sandboxing stops being optional and starts being table stakes.

### Isolation options compared

| Option | What it isolates | What it’s good at | Big constraint | My default |
| --- | --- | --- | --- | --- |
| Web Worker | CPU work off main thread | inference/planning, parsing | shares origin privileges via messages if you’re sloppy | Yes |
| sandboxed iframe | DOM separation | rendering untrusted UI | still same-device, `postMessage` complexity | Sometimes |
| WebAssembly (Wasm) sandbox | memory + execution model | running untrusted compute code | you still must control imports/egress | Yes |
| WebContainers | Node.js-like runtime in-browser | running npm-ish toolchains | requires COOP/COEP + `SharedArrayBuffer` | Only for power users |

WebContainers specifically require `SharedArrayBuffer`, which requires cross-origin isolation with COOP/COEP headers. That’s straight from the [WebContainers quickstart](https://webcontainers.io/guides/quickstart).

If you deploy a typical SPA behind a bunch of third-party tags, cross-origin isolation is not a freebie. Expect to spend time on headers and embedding constraints.

Also, don’t confuse “Wasm sandbox” with “secure by default.” Wasm reduces some classes of memory corruption, but you can still exfiltrate data if you hand it network access.

For a deeper on-device hardening mindset, see [How to secure local LLM inference](/blog/secure-local-llm-inference).

## Threat model: web-native agents (OWASP LLM + classic web threats)

Browser agents combine two threat families: LLM-specific attacks and classic web app attacks.

This is why I like the OWASP taxonomy. The OWASP Foundation Top 10 gives you language to talk about prompt injection and insecure output handling without hand-waving.

Threats you must assume:

- **Prompt injection (direct + indirect):** untrusted content tells the agent to do something else.
- **XSS:** attacker runs JS in your origin, reads memory, triggers tool calls.
- **Ambient authority abuse:** tool calls inherit cookies/session and can act as the user.
- **Data exfiltration:** the agent “helpfully” sends secrets to a remote endpoint.
- **Supply chain:** model weights/tool bundles swapped or poisoned.
Mitigations that actually ship:

- CSP that blocks inline scripts. Treat this like a banking app.
- Strict tool allowlists and schema validation in the service worker.
- Confirmation for high-risk tools with transaction previews.
- Separate storage buckets for memory vs caches so eviction hurts less.
- Integrity checks for model assets. Hash and verify.
If you want a prompt injection deep dive, I’ve already written [How to stop repo prompt injection in coding agents](/blog/repository-prompt-injection-coding-agent) and [Indirect prompt injection in AI agents](/blog/indirect-prompt-injection-ai-agents).

[Image: Threat model checklist — injection, XSS, exfiltration, supply chain, confirmations]

### How can I prevent prompt injection in tool-using agents?

You don’t “prevent” it with a better prompt. You contain it with architecture.

- Treat all external text as hostile.
- Never let the model execute tools without policy checks.
- Require confirmations where harm is possible.
If you do those three, prompt injection becomes annoying instead of catastrophic.

## What this means next

Browser-native agents are going to blow up because they’re the first agent UX that feels like software again. Low latency. Local state. Offline mode. No per-token tax for every keystroke.

My prediction for 2027 is simple. The winners won’t be the agents with the fanciest models. They’ll be the ones with the most boring architecture: service-worker tool bridges, capability scopes, eviction-aware memory, and confirmations that normal users actually understand.

If you’re building a browser agent, do one thing this week: move tool execution behind a service worker and add an allowlist. If that feels like overkill, you’re building a demo, not a product.

Photo by Rahul Mishra on Unsplash.

## FAQ

### Can you run an LLM completely in the browser?

Yes. Modern in-browser runtimes can run smaller models using WebGPU or WebAssembly, which keeps data local and can work offline. The trade-off is performance and memory limits compared to calling a remote API. Many teams ship a hybrid mode: local for privacy/offline, remote for heavier tasks.

### Is IndexedDB persistent, and how much can it store?

IndexedDB is persistent in the sense that it survives reloads, but it’s not guaranteed permanent. Browsers enforce storage quotas and may evict site data when the device is under storage pressure, and users can also clear it manually. Design agent memory with compaction and export/backup options rather than assuming unlimited space.

### How do service workers intercept requests for an AI agent tool bridge?

A service worker sits between your web app and the network and can intercept fetch requests made by pages it controls. For an agent, you can route tool calls through the service worker so it can enforce domain allowlists, validate parameters, require user confirmation for high-risk actions, and record a local audit trail.