The mental model you need is simple: the chat completions API is stateless. It receives a list of messages and returns one more message. Your job in the tool-call loop is to manage that list correctly across multiple round trips. When the model wants to call a tool, it returns a message with finish_reason == "tool_calls" and a tool_calls array instead of content. That message goes onto the list. You run each tool, wrap each result in a message with role: tool and the matching tool_call_id, append those too, then call the API again. The model now sees its own tool call and the result sitting right next to each other in history, which is how it knows what happened and what to do next. Nothing is stored server-side between calls.
A real scenario: a user asks "Book me the cheapest flight from NYC to London next month and add it to my calendar." That requires at least three tool calls in sequence -- search flights, pick the best result, then create a calendar event with the flight details. The model cannot do any of those things in a single shot. It has to call search_flights, wait for your code to hit Amadeus or Skyscanner, read the returned options, call book_flight with the chosen itinerary, read the confirmation number, then call create_calendar_event. A senior engineer would sketch the message array on paper before writing code: what does messages look like after turn 1, after turn 2, after turn 3? Anything unexpected in that list will produce confused model behavior that is very hard to debug from the output alone.
Compared to pure prompt chaining (where you pipe outputs between separate LLM calls), the tool-call loop keeps everything in one conversation context. That means the model can refer back to earlier tool results, correct itself if a later tool call contradicts an earlier one, and decide mid-task that it does not need to call the next tool at all. The downside is that every turn of the loop costs tokens for the full message history. Prompt chaining lets you trim and summarize between steps, which matters when tasks get long. For tasks under roughly 10 tool calls on a 128k-context model, the single-context loop is simpler and usually preferred. For very long-running tasks -- think an agent that processes hundreds of documents -- you will need to either summarize history or split into chained sub-agents.
At 10 users, none of this matters much. You run the loop synchronously, it takes a few seconds, users wait. At 10k concurrent users, each running an agent loop with 5 round trips at 500ms per trip, you are looking at 2.5 seconds of sequential latency per user and a pile of concurrent HTTP connections to the model provider. You need async execution of tool calls when the model requests multiple tools in one turn (the tool_calls array can have more than one entry), and you want to parallelize independent tool calls with asyncio.gather. You also need per-user loop state so concurrent sessions do not collide. At 10M users you almost certainly need a queue-based architecture: the loop state is serialized and stored (Redis, a database), a worker picks it up, runs one turn, writes the result back, and the next worker continues. This makes the loop resumable across crashes and horizontally scalable.
Cost and latency interact in a specific way in the loop. Every turn re-sends the full history, so a 5-turn loop with 500 tokens per turn does not cost 5 x 500 = 2500 input tokens -- it costs 500 + 1000 + 1500 + 2000 + 2500 = 7500 input tokens because each call includes all prior turns. At GPT-4o pricing this is illustrative but non-trivial at scale. You can trim stale tool results from history once the model has processed them, replacing the raw JSON with a short summary string. The model already incorporated that information; keeping 2KB of raw API response in every subsequent turn is just wasted tokens. Finally, always set a max_iterations guard. A model in a confused state will sometimes keep calling the same tool in a loop. Without a cap, that will run until you hit a rate limit or a bill arrives.
Key Takeaways
- The model emits tool calls as structured data; your code executes them and returns results.
- Every tool result must be appended as a
toolrole message before the next API call. - Add a max-iterations cap to every loop or a runaway agent will burn your budget.
- The full conversation history, including tool results, is what the model reasons over each turn.
Pro tips
- When the model returns multiple tool calls in one turn, check whether they are independent -- if they are, run them with asyncio.gather and cut wall-clock latency proportionally. Sequential execution of parallel-safe calls is the single most common performance mistake in agent code.
- Trim raw tool results from history once they are two or more turns old. Replace the full JSON blob with a one-line summary string. The model already incorporated the data; leaving 3KB of raw response in every subsequent prompt is wasted input tokens and degrades reasoning on long tasks.
- Log the full message array to a structured store (not just stdout) before every API call. When an agent misbehaves in production, the message history at the moment of failure is the only artifact that tells you exactly what the model saw. Without it you are guessing.
- Never trust that the model will stop. Always set a hard MAX_ITERATIONS limit and return a graceful degradation message rather than an exception. An agent that terminates cleanly with a partial answer is far better than one that crashes or loops until it burns rate-limit quota.
Common pitfalls
- Mistake: Forgetting to append the assistant's tool-call message before the tool result messages. Fix: Always push
msg(the assistant turn) ontomessagesimmediately after receiving the response, before appending any tool results. - Mistake: Matching tool results to calls by position instead of
tool_call_id. Fix: Usetc.idas thetool_call_idfield in every tool-result message; position-based matching breaks when the model emits multiple calls. - Mistake: Letting tool exceptions propagate and crash the loop. Fix: Catch exceptions inside your executor, return a JSON error payload as the tool result, and let the model decide how to handle the failure gracefully.
- Mistake: Ignoring context window growth across many loop iterations. Fix: Track cumulative input tokens per loop; summarize or prune old tool results once total history exceeds roughly 60 percent of the model's context limit.
When to use a single-context tool-call loop vs prompt chaining
| Option | Use when | Avoid when |
|---|---|---|
| Single-context tool-call loop | Task needs fewer than ~10 tool calls, model must refer back to earlier results, or you want the simplest possible implementation. | Task requires hundreds of steps; accumulated history would overflow the context window or cost too much in input tokens. |
| Prompt chaining between separate LLM calls | You need to summarize or filter intermediate results before the next step, or each sub-task is logically independent and history isolation is safer. | The later steps need rich context from earlier tool results; chaining forces you to serialize that context manually. |
| Queue-backed resumable loop | Agent runs are long-lived (minutes to hours), need to survive worker restarts, or must scale horizontally across many concurrent users. | Low-volume, short tasks where the overhead of serializing state to a store outweighs the benefit. |
Code Example
# openai>=1.0.0
import json, openai
client = openai.OpenAI()
def get_weather(location: str) -> dict:
return {"location": location, "temp_c": 18, "condition": "cloudy"}
tools = [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Return current weather for a city.",
"parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]}
}
}]
messages = [{"role": "user", "content": "What is the weather in Tokyo?"}]
while True:
resp = client.chat.completions.create(model="gpt-4o-mini", tools=tools, messages=messages)
msg = resp.choices[0].message
messages.append(msg) # always append the assistant turn
if msg.tool_calls is None:
print(msg.content)
break
for tc in msg.tool_calls:
result = get_weather(**json.loads(tc.function.arguments))
messages.append({"role": "tool", "tool_call_id": tc.id, "content": json.dumps(result)})How this code works
This code demonstrates an AI "tool-call loop," where a model dynamically decides to use a function, executes it, and then incorporates the result to continue a conversation. Starting with a user asking about "What is the weather in Tokyo?", the system uses an AI model (gpt-4o-mini) to understand the intent. The core idea is for the AI to autonomously determine when to call a predefined Python function, get_weather, to fulfill the user's request. The tools list provides the AI with a structured description of get_weather, including its purpose and required parameters.
The while True loop is where the magic happens. In each iteration, client.chat.completions.create sends the ongoing messages history to the AI. If the AI decides it needs to use get_weather, its response (msg) will contain tool_calls. The code then extracts the location argument specified by the AI using json.loads(tc.function.arguments), calls the actual get_weather Python function, and appends the result back to messages with a role: "tool". A subtle but crucial step is messages.append(msg) immediately after receiving the AI's response; this ensures the AI's own tool_calls instruction is added to the conversation history before the tool's output, maintaining the correct chronological flow for the model to process in the next loop iteration. The loop continues until the AI provides a final text content response without tool_calls, at which point it prints the answer and breaks.
Production-grade example
Adds retries with backoff, async parallel tool execution, per-iteration token logging, max-iterations guard, and graceful error returns.
# openai>=1.0.0, tenacity>=8.0
import json, logging, os, time, asyncio
from typing import Any
import openai
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger(__name__)
client = openai.AsyncOpenAI(api_key=os.environ["OPENAI_API_KEY"])
MAX_ITERATIONS = 10
MODEL = "gpt-4o-mini"
async def get_weather(location: str) -> dict:
await asyncio.sleep(0.05) # simulate real HTTP call
return {"location": location, "temp_c": 18, "condition": "cloudy"}
TOOL_REGISTRY: dict[str, Any] = {"get_weather": get_weather}
@retry(
retry=retry_if_exception_type((openai.RateLimitError, openai.APITimeoutError)),
wait=wait_exponential(multiplier=1, min=2, max=30),
stop=stop_after_attempt(4),
)
async def call_model(messages: list, tools: list) -> openai.types.chat.ChatCompletion:
return await client.chat.completions.create(
model=MODEL, tools=tools, messages=messages, timeout=30
)
async def run_agent(user_message: str, tools: list) -> str:
messages = [{"role": "user", "content": user_message}]
total_input_tokens = 0
total_output_tokens = 0
for iteration in range(MAX_ITERATIONS):
log.info("iteration=%d messages=%d", iteration, len(messages))
try:
resp = await call_model(messages, tools)
except openai.BadRequestError as exc:
log.error("bad_request error=%s", exc)
return "Agent encountered an unrecoverable error."
total_input_tokens += resp.usage.prompt_tokens
total_output_tokens += resp.usage.completion_tokens
msg = resp.choices[0].message
messages.append(msg)
if msg.tool_calls is None:
log.info("done input_tokens=%d output_tokens=%d", total_input_tokens, total_output_tokens)
return msg.content
# Execute all tool calls concurrently
async def execute_one(tc):
fn = TOOL_REGISTRY.get(tc.function.name)
if fn is None:
return tc.id, json.dumps({"error": f"unknown tool {tc.function.name}"})
try:
args = json.loads(tc.function.arguments)
result = await fn(**args)
log.info("tool=%s args=%s", tc.function.name, args)
return tc.id, json.dumps(result)
except Exception as exc: # noqa: BLE001
log.warning("tool_error tool=%s error=%s", tc.function.name, exc)
return tc.id, json.dumps({"error": str(exc)})
results = await asyncio.gather(*[execute_one(tc) for tc in msg.tool_calls])
for tool_call_id, content in results:
messages.append({"role": "tool", "tool_call_id": tool_call_id, "content": content})
log.warning("max_iterations_reached limit=%d", MAX_ITERATIONS)
return "Agent reached the maximum number of steps without completing the task."How this code works
This code implements an AI agent capable of using external tools to fulfill complex requests from a user, mimicking a loop where the AI decides what to do, the program executes it, and the AI then processes the result. The main run_agent function manages this conversational turn-taking, starting with a user_message, sending it to the MODEL via call_model, and continuously updating the messages list with both the AI's responses and the outcomes of any tool usage. This iterative process continues until the model provides a final text response or a set limit is reached.
The TOOL_REGISTRY holds the available functions, like get_weather, that the AI can call. When the model responds with msg.tool_calls, the agent executes these functions. Importantly, asyncio.gather is used to run multiple tool calls at the same time for efficiency, and their results are then added back to the messages list as {"role": "tool"} messages, ensuring the model has the full context for its next turn. A subtle but critical feature is MAX_ITERATIONS, which prevents the agent from getting stuck in an endless loop by limiting the number of steps it can take before providing an answer or declaring it cannot complete the task.
Practice & master
Try the exercise, check your understanding, then mark this lesson mastered to track your path to pro.
Exercise
Build a simple agent loop that answers the question 'What is the population of France and what is 10 percent of it?' using two tools: get_population(country) and calculate_percent(value, percent). The agent should call both tools across one or more loop iterations and return a final natural-language answer. Add a max-iterations guard of 5.
import json
import openai
client = openai.OpenAI()
# TODO: implement these two stub functions
def get_population(country: str) -> dict:
pass # return {"country": country, "population": <int>}
def calculate_percent(value: float, percent: float) -> dict:
pass # return {"result": <float>}
tools = [
# TODO: define tool schemas for both functions
]
TOOL_REGISTRY = {"get_population": get_population, "calculate_percent": calculate_percent}
MAX_ITERATIONS = 5
def run_agent(user_message: str) -> str:
messages = [{"role": "user", "content": user_message}]
for _ in range(MAX_ITERATIONS):
resp = client.chat.completions.create(model="gpt-4o-mini", tools=tools, messages=messages)
msg = resp.choices[0].message
messages.append(msg)
# TODO: check if the model returned tool calls or a final answer
# TODO: execute each tool call and append results
pass
return "Max iterations reached."
print(run_agent("What is the population of France and what is 10% of it?"))Quick check
After the model returns a tool-call response, what must happen before you call the API again?
Why does the input token count grow with each iteration of the tool-call loop?
A model in a buggy state keeps calling the same tool repeatedly. What is the correct mitigation?
messages array after two turns of a loop where the model calls one tool in turn one and two tools simultaneously in turn two. How many messages are in the array at the start of turn three, and what role is each one?