The mental model: specialized agents as microservices
Think of each agent as a microservice with a language interface instead of an HTTP interface. A researcher agent exposes one operation: given a topic, return a structured summary. A code agent exposes one operation: given a spec, return working code. An orchestrator agent composes these operations in sequence or in parallel, passing outputs as inputs. The critical insight is that the boundary between agents is a message, not a function call, which means you can checkpoint, retry, audit, and replace individual agents without touching the rest of the system.
Under the hood, most multi-agent frameworks model the system as a directed graph. Nodes are agents; edges are message flows. LangGraph makes this explicit with a state machine. CrewAI uses a sequential or hierarchical process model. AutoGen uses an actor model where agents respond to each other's messages. The orchestrator pattern is the safest for production: one agent holds the plan and delegates, so you have a single place to enforce budget limits, timeouts, and safety checks. Fully decentralized agent meshes where agents call each other freely look elegant on a whiteboard but are hard to debug when something goes wrong at step 17 of 23.
A real-world scenario: competitive intelligence pipeline
Imagine you're building a weekly competitive intelligence report. A single-agent approach fails here because the task requires web search, financial data lookup, natural language summarization, structured data extraction, and final synthesis. That's too many tools and too much context for one agent to handle reliably.
A pro would decompose it like this: a Planner agent takes the target company name and outputs a list of sub-tasks with dependencies. A Search agent runs web searches for each sub-task and returns raw snippets. An Extraction agent parses those snippets into structured JSON (company name, metric, date, source URL). An Analysis agent compares this cycle's data against last cycle's stored data. A Synthesis agent writes the final Markdown report. An optional Critic agent runs a quick factual consistency check before delivery.
You'd wire these with a message-passing layer (Redis streams or a simple in-memory queue for local dev), give the Orchestrator a budget of N total LLM calls, and log every agent's input/output pair for debugging. If the Search agent returns no results for a sub-task, the Orchestrator retries with a rephrased query or marks that section as "data unavailable" rather than hallucinating.
Tradeoffs vs. alternatives
The main alternative to multi-agent decomposition is a single powerful agent with a large context window. GPT-4o and Claude 3.5 Sonnet both support 128k+ tokens, which is tempting. The problem is that long contexts increase latency, cost, and the probability of the model losing track of details buried in the middle of the prompt (the "lost in the middle" effect). A single agent also serializes everything; you cannot parallelize independent sub-tasks.
The other alternative is rigid prompt chaining without agents at all. This works fine for linear pipelines with deterministic steps, but falls apart when you need conditional branching, error recovery, or dynamic task generation. Agents add autonomy: they can decide what tool to call next based on intermediate results. The tradeoff is observability and control. Pure prompt chains are easier to trace; agent systems require explicit logging and tracing infrastructure from day one.
What changes at scale
At 10 users, you can run a multi-agent crew synchronously in a single process. At 10k users, each crew run needs to be async, queued, and isolated. You need per-run state management (a database row per run, not in-memory dicts), and you need to think about agent timeout budgets so one slow search agent doesn't block the whole pipeline. At 10M users, the economics of model selection become critical: the researcher agent running gpt-4o on every request at scale will bankrupt you. You'd route simple extraction tasks to gpt-4o-mini or a fine-tuned smaller model, and reserve gpt-4o for the synthesis step that actually requires reasoning depth. You'd also add caching at the tool layer so repeated lookups hit a cache before the LLM even sees them.
Cost, latency, and reliability
Latency is the hardest problem. A 5-agent sequential pipeline where each agent averages 3 seconds adds up to 15 seconds minimum, which is unacceptable for synchronous UI. The solution is to identify which tasks have no dependencies and run those agents in parallel using asyncio. An Orchestrator that fans out to 3 parallel agents and waits for all three reduces wall-clock time to the longest individual task. Build your graph before you build your agents; the dependency structure determines your latency floor. For reliability, treat agent failures as expected events. Every agent call should have a timeout, a retry budget, and a fallback behavior (return empty result, use cached data, or escalate to human review).
Key Takeaways
- Assign each agent exactly one job and a minimal tool set to prevent context bloat.
- Use an orchestrator agent to route tasks; never let agents call each other directly without governance.
- Pick the cheapest model that meets the quality bar for each specialized agent role.
- Design explicit message contracts between agents so failures are observable and retryable.
Pro tips
- Give each agent a role budget constraint in its system prompt: 'You have at most 3 tool calls to answer this.' Without a budget, agents loop on ambiguous tasks and burn tokens. An explicit cap forces the agent to commit to an answer with available information.
- Log every inter-agent message as a structured event with a shared trace ID. When a multi-agent run produces a bad result, you need to replay the exact message sequence that caused it. Without trace IDs threading through all agent calls, debugging a 6-agent pipeline is close to impossible.
- Use different models for different agent roles based on what the role actually demands. A routing or extraction agent rarely needs GPT-4o reasoning ability; gpt-4o-mini at one-tenth the cost does the job. Reserve your expensive model budget for the synthesis or critic agent where reasoning depth matters.
- Design your orchestrator to accept a 'partial results' mode. If the search agent times out, the pipeline should synthesize from whatever data it has and flag the output as incomplete rather than failing the whole run. Users can tolerate incomplete answers; they cannot tolerate errors.
Common pitfalls
- Mistake: Letting agents pass raw unvalidated output directly to the next agent. Fix: Define a Pydantic schema for every inter-agent message and validate before passing. One malformed output cascades into garbage downstream.
- Mistake: Building a circular dependency between agents where agent A waits on B and B waits on A. Fix: Draw the dependency graph before writing code. Every edge must point in one direction; cycles require an explicit arbitration mechanism.
- Mistake: Using the same large context window LLM for every agent role because it's easier. Fix: Profile the token usage per agent role after your first prototype. Reassign cheap roles to smaller models before costs compound.
- Mistake: Treating multi-agent pipelines as synchronous request-response from the start. Fix: Design for async from day one with asyncio or a task queue (Celery, ARQ). Retrofitting async into a synchronous pipeline is significantly more painful than building it right initially.
When to use multi-agent vs single-agent vs prompt chaining
| Option | Use when | Avoid when |
|---|---|---|
| Single agent with tools | Task fits in one context window, requires fewer than ~5 tool calls, and has no parallel sub-tasks. | Task needs multiple distinct skill sets or the prompt becomes longer than ~4k tokens of instructions. |
| Sequential prompt chaining | Steps are deterministic and linear; no branching, no error recovery, no dynamic task generation needed. | Steps have dependencies on intermediate results that require conditional routing or retries. |
| Multi-agent with orchestrator | Task decomposes into parallel sub-tasks, requires different model capabilities per step, or needs independent failure boundaries. | You need sub-second latency or the overhead of agent coordination outweighs the complexity of the task. |
| Fully decentralized agent mesh | Each agent is fully autonomous and can self-organize; research or simulation contexts with relaxed correctness requirements. | Production systems where observability, cost control, and deterministic routing are required. |
Code Example
# crewai==0.28.0
from crewai import Agent, Task, Crew
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
researcher = Agent(
role="Researcher",
goal="Find factual information about the given topic",
backstory="Expert at distilling web information into concise summaries",
llm=llm,
verbose=False,
)
writer = Agent(
role="Writer",
goal="Turn research notes into a clear, structured report",
backstory="Technical writer with experience in AI topics",
llm=llm,
verbose=False,
)
research_task = Task(
description="Research the current state of LLM context windows in 2024",
expected_output="Bullet-point summary of key facts and numbers",
agent=researcher,
)
write_task = Task(
description="Write a 200-word report using the research notes",
expected_output="A structured 200-word report",
agent=writer,
context=[research_task],
)
crew = Crew(agents=[researcher, writer], tasks=[research_task, write_task])
result = crew.kickoff()
print(result)How this code works
This code demonstrates how to build a collaborative multi-agent system using crewai, where specialized AI agents work together to produce a structured report. It sets up a small team to research a topic and then write about it, showcasing a foundational pattern for dividing complex problems among AI assistants.
The process begins by defining an llm (a language model) that all agents will use as their intelligence. Then, two Agents are created: a researcher and a writer. Each agent is given a specific role, a clear goal (what it needs to achieve), and a backstory to define its persona and capabilities. Setting verbose=False keeps the output clean by hiding the agents' internal thought processes. Next, Tasks are defined. The research_task is assigned to the researcher to gather information, specifying its description and expected_output. The write_task is then assigned to the writer, but critically, it includes context=[research_task]. This subtle detail ensures the writer waits for and uses the output of the research_task before starting its own work, orchestrating the collaboration and execution order. Finally, a Crew is assembled with both agents and tasks, and crew.kickoff() starts the entire workflow, producing the final report.
Production-grade example
Adds per-agent retries with backoff, timeouts, token logging, graceful degradation on agent failure, and async parallel execution.
# langgraph==0.1.13, langchain-openai==0.1.8
import asyncio
import logging
import os
import time
from typing import TypedDict, Annotated
import operator
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage, SystemMessage
from langchain_core.exceptions import OutputParserException
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
from openai import RateLimitError, APITimeoutError
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(name)s %(levelname)s %(message)s")
logger = logging.getLogger("multi_agent")
researcher_llm = ChatOpenAI(
model="gpt-4o-mini",
temperature=0,
timeout=20,
api_key=os.environ["OPENAI_API_KEY"],
)
analyst_llm = ChatOpenAI(
model="gpt-4o",
temperature=0.2,
timeout=30,
api_key=os.environ["OPENAI_API_KEY"],
)
@retry(
retry=retry_if_exception_type((RateLimitError, APITimeoutError)),
wait=wait_exponential(multiplier=1, min=2, max=30),
stop=stop_after_attempt(4),
)
async def call_agent(llm, system_prompt: str, user_message: str, agent_name: str) -> str:
start = time.monotonic()
try:
response = await llm.ainvoke([
SystemMessage(content=system_prompt),
HumanMessage(content=user_message),
])
elapsed = time.monotonic() - start
usage = response.response_metadata.get("token_usage", {})
logger.info(
"agent_call",
extra={
"agent": agent_name,
"input_tokens": usage.get("prompt_tokens", 0),
"output_tokens": usage.get("completion_tokens", 0),
"latency_s": round(elapsed, 3),
},
)
return response.content
except (RateLimitError, APITimeoutError):
logger.warning("Retryable error in agent %s, backing off", agent_name)
raise
except Exception as exc:
logger.error("Non-retryable failure in agent %s: %s", agent_name, exc)
return "[agent_error: no output available]"
async def run_pipeline(topic: str) -> dict:
RESEARCHER_PROMPT = "You are a research assistant. Return 5 concise bullet points of key facts."
ANALYST_PROMPT = "You are a senior analyst. Given research notes, produce a 150-word executive summary with a risk assessment."
research_notes, _ = await asyncio.gather(
call_agent(researcher_llm, RESEARCHER_PROMPT, topic, "researcher"),
asyncio.sleep(0),
)
if "agent_error" in research_notes:
logger.warning("Researcher failed, falling back to minimal context for analyst")
research_notes = f"Limited data available for: {topic}"
summary = await call_agent(
analyst_llm,
ANALYST_PROMPT,
f"Research notes:\n{research_notes}",
"analyst",
)
return {"topic": topic, "research": research_notes, "summary": summary}
if __name__ == "__main__":
result = asyncio.run(run_pipeline("Open-source LLM adoption in enterprise 2024"))
print(result["summary"])How this code works
This code showcases a multi-agent architecture where specialized AI agents collaborate to process information. It defines two distinct roles: a researcher_llm and an analyst_llm. The researcher_llm is designed to gather key facts about a given topic, while the analyst_llm then synthesizes these facts into an executive summary, including a risk assessment. This sequential workflow demonstrates how different AI capabilities can be chained to accomplish a more complex task than any single agent could perform alone.
The process begins by setting up two ChatOpenAI instances with different models (gpt-4o-mini for the researcher, gpt-4o for the analyst) and temperature settings to reflect their specialized functions. The call_agent function is a robust wrapper that handles communication with the LLMs, injecting SystemMessage and HumanMessage prompts. A key feature here is the @retry decorator, which automatically retries the call if transient network issues like RateLimitError or APITimeoutError occur, making the system more resilient. The run_pipeline function orchestrates the interaction: it first calls the researcher agent, and then uses the research_notes output to inform the analyst agent. A subtle but important detail is the if "agent_error" in research_notes: check. If the researcher agent fails to produce output, the system doesn't crash but instead provides minimal context to the analyst, ensuring the pipeline can still deliver a result, albeit a degraded one.
Practice & master
Try the exercise, check your understanding, then mark this lesson mastered to track your path to pro.
Exercise
Build a two-agent research pipeline using CrewAI or direct LLM calls. Agent 1 (Researcher) takes a company name and returns 3 bullet points of recent facts. Agent 2 (Critic) reviews those facts and flags any that appear vague or unverifiable. Wire them sequentially so the Critic always sees the Researcher's output.
# crewai==0.28.0 or use direct ChatOpenAI calls
import os
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage, SystemMessage
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0,
api_key=os.environ["OPENAI_API_KEY"])
RESEARCHER_SYSTEM = "You are a research assistant. Return exactly 3 bullet points of recent, specific facts."
CRITIC_SYSTEM = "You are a fact-checking critic. For each bullet point, label it VERIFIED, VAGUE, or UNVERIFIABLE and explain why in one sentence."
def researcher_agent(company: str) -> str:
# TODO: call llm with RESEARCHER_SYSTEM and the company name
pass
def critic_agent(research_notes: str) -> str:
# TODO: call llm with CRITIC_SYSTEM and pass research_notes as user message
pass
def run_pipeline(company: str) -> None:
# TODO: call researcher_agent, print output, then call critic_agent with result, print output
pass
run_pipeline("Anthropic")Quick check
Why does assigning each agent a minimal, focused tool set improve multi-agent system reliability?
An orchestrator agent delegates to 3 parallel sub-agents. Sub-agent 2 times out. What is the correct production behavior?
You have a 6-agent sequential pipeline where each agent averages 4 seconds. What is the most effective way to reduce total latency to under 10 seconds?