Retry logic is the first thing every developer reaches for, and it's also the first thing most developers get wrong. The naive version -- retry three times with a 1-second sleep -- is almost useless against real-world API failures. When an LLM provider's rate limiter fires, all of your concurrent workers hit the 429 at the same millisecond and then all retry at t+1s. You've just created a thundering herd that hits the rate limiter again immediately. Exponential backoff (wait 2s, then 4s, then 8s) spreads the retries out over time. Jitter (add random(0, backoff_seconds)) scatters concurrent workers so they don't all wake up together. tenacity gives you both with one decorator. The key parameter to tune is stop_after_attempt -- retrying more than 4-5 times on a 429 usually means you need a different solution (circuit breaker, fallback, or provisioned throughput), not more retries.
Fallbacks handle the case where retries eventually exhaust. The mental model: your primary path is the expensive, high-quality option; your fallback path is the cheaper, lower-latency, or locally-available option. A concrete scenario -- you're building a document analysis workflow that calls GPT-4o to extract structured data. The API is down for 8 minutes due to an incident. Retries exhaust after 30 seconds. Your fallback: run the same extraction prompt against a self-hosted Llama 3.1 70B endpoint that's slower and less accurate, but available. The fallback response gets flagged with a confidence: low field so downstream systems know to treat it differently. Fallbacks don't need to be equivalent in quality -- they just need to keep the workflow alive and make the degradation visible.
Circuit breakers solve the problem that neither retries nor fallbacks address: a dependency that is overloaded and needs relief, not more traffic. The state machine has three states. Closed: all requests go through normally. Open: the breaker has tripped (failure rate exceeded threshold), all requests fail fast without touching the dependency. Half-open: after a recovery timeout, one test request gets through. If it succeeds, transition back to closed. If it fails, go back to open and reset the timer. The practical effect is that a struggling downstream service gets a window to recover while your workflow fails fast instead of queueing up thousands of requests that will all time out. Libraries like circuitbreaker or pybreaker implement this state machine; you can also implement it manually with a Redis key tracking the open/closed state (which lets multiple workers share circuit state).
The real-world scenario where all three patterns combine: a multi-agent research workflow that orchestrates web search, a vector database retrieval step, and two sequential LLM calls. Under normal load, all three patterns are invisible -- they never fire. Under a web search API outage, the circuit breaker opens after 5 consecutive failures, and the fallback switches to a cached web search index. Under an LLM provider rate limit, exponential backoff retries absorb the 429s until the token bucket refills. Under a vector database timeout, retries fail after 3 attempts, the fallback returns an empty retrieval result, and the generation step is instructed to answer from its own knowledge with reduced confidence. None of these failures cascade into a total workflow halt.
At scale, the tradeoffs shift. At 10 users, retries with 30s max backoff are fine -- the latency tail is acceptable and concurrent retry storms don't exist. At 10k users, exponential backoff with jitter is mandatory, and you'll want circuit breakers per-dependency with shared state (Redis) so all workers see the same circuit status. You also need bulkheads: separate thread pools or async semaphores per dependency, so a slow LLM API doesn't exhaust the connection pool for your fast vector database. At 10M users, you're adding provider-level redundancy (OpenAI and Anthropic as parallel primaries, not just fallbacks), shadow traffic to warm fallback providers, and SLO-based circuit breaker thresholds that tie into your alerting stack. Cost implications are significant at scale: every retry is a duplicate API call and duplicate token cost. Log retry counts per request and set up an alert if your retry rate exceeds 5% of traffic -- that's a signal your quotas or your code have a structural problem, not a transient one.
Key Takeaways
- Use exponential backoff with jitter on retries to avoid thundering-herd amplification against rate-limited APIs.
- Fallbacks degrade gracefully -- serve a cached response or cheaper model rather than returning a 500.
- Circuit breakers protect a struggling dependency by stopping traffic before it collapses under retry storms.
- Classify every failure as transient or persistent first; that classification drives which pattern to apply.
Pro tips
- Never retry on 4xx errors except 429 and 408. A 400 (bad request) or 401 (bad auth) will always fail -- retrying it burns quota and adds latency for no reason. Filter your retry predicate to specific status codes or exception types.
- Pass a
Retry-Afterheader value as the wait time when the API returns one. Providers like OpenAI and Anthropic include it on 429 responses. Ignoring it and using your own backoff is technically correct but wastes time when the provider tells you exactly when capacity opens up. - Separate your circuit breaker state from your process. A per-process in-memory circuit breaker means each of your 20 worker pods independently trips, recovers, and re-trips -- they never coordinate. Store the state in Redis and all workers share a single view of whether the dependency is healthy.
- Model your fallback response structure to match your primary. If your primary returns a structured JSON object with a
confidencefield, your fallback should return the same shape withconfidence: low. Downstream code that assumes the primary shape will break silently otherwise.
Common pitfalls
- Mistake: Retrying non-idempotent operations like charge-a-card or send-an-email. Fix: Only retry reads and idempotent writes. Use an idempotency key on write operations so the server deduplicates if the retry fires after a partial success.
- Mistake: Setting
stop_after_attemptto 10+ on LLM calls. Fix: Cap at 3-4 attempts. Beyond that you're masking a systemic quota or reliability problem that needs a fallback or provisioned throughput, not more retries. - Mistake: Using a single circuit breaker for all dependencies in a workflow. Fix: One circuit breaker per external dependency. An open circuit on your web search tool should not block LLM calls.
- Mistake: Silently swallowing fallback responses without marking them as degraded. Fix: Always tag fallback responses with a
degraded: truefield and log the reason. Downstream systems and monitoring dashboards need to know the response quality is reduced.
When to use retries vs fallbacks vs circuit breakers
| Option | Use when | Avoid when |
|---|---|---|
| Retry with backoff | Failure is transient -- 429 rate limit, network blip, 503 under momentary load spike. | Failure is persistent or structural (auth error, malformed request, dependency is down for minutes). |
| Fallback | Primary path is unavailable or too slow but an alternate path (cheaper model, cache, stub) can provide acceptable output. | The fallback output is so degraded it would mislead downstream steps; better to fail fast and surface the error. |
| Circuit breaker | A dependency has high sustained failure rates and continuing to call it wastes resources or makes the outage worse. | You have a single-user, single-process system where shared circuit state adds complexity for no benefit. |
| All three combined | Production multi-agent workflows calling external APIs, LLM providers, vector DBs under real traffic. | Prototyping, local scripts, or internal calls to services you control with SLAs enforced by your own team. |
Code Example
# tenacity==8.2.3
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
import openai
@retry(
retry=retry_if_exception_type((openai.RateLimitError, openai.APITimeoutError)),
wait=wait_exponential(multiplier=1, min=2, max=30),
stop=stop_after_attempt(4),
)
def call_llm(prompt: str) -> str:
response = openai.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
timeout=15,
)
return response.choices[0].message.content
result = call_llm("Summarize this paragraph in one sentence.")
print(result)How this code works
This code demonstrates how to make calls to an AI model, specifically OpenAI's gpt-4o-mini, more resilient by automatically handling temporary failures. In advanced AI workflow orchestration, ensuring reliability is crucial, so this example uses the tenacity library to configure automatic retries for API requests that might fail due to transient issues. The call_llm function wraps the actual openai.chat.completions.create call, and the core of its failure handling is the @retry decorator applied above it, which dictates when and how retries should occur instead of letting the program crash on the first transient error.
The @retry decorator is configured with specific rules. retry_if_exception_type tells tenacity to only retry if it encounters an openai.RateLimitError (indicating too many requests) or an openai.APITimeoutError (indicating the API took too long). wait_exponential implements a backoff strategy, progressively increasing the delay between retries, starting at 2 seconds and capping at 30 seconds, to avoid overwhelming the API. A subtle but important detail is stop_after_attempt(4), which means the function will try a maximum of four times in total—one initial attempt plus three retries. The openai.chat.completions.create call itself also includes a timeout=15 parameter, setting a 15-second limit for each individual attempt before an openai.APITimeoutError is raised, potentially triggering one of tenacity's retries.
Production-grade example
Circuit breaker plus retry plus fallback, with structured cost logging and explicit degradation signaling.
# tenacity==8.2.3 pybreaker==1.2.0 structlog==24.1.0
import os, time, random, structlog
from tenacity import (
retry, stop_after_attempt, wait_exponential_jitter,
retry_if_exception_type, before_sleep_log, RetryError,
)
import pybreaker
import openai
log = structlog.get_logger()
# Shared circuit breaker: trips after 5 failures, resets after 60s
_breaker = pybreaker.CircuitBreaker(fail_max=5, reset_timeout=60)
openai_client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])
def _primary_call(prompt: str, model: str = "gpt-4o") -> str:
"""Raw LLM call wrapped by the circuit breaker."""
@_breaker
def _inner():
return openai_client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
timeout=20,
)
resp = _inner()
log.info("llm_call_success", model=model,
prompt_tokens=resp.usage.prompt_tokens,
completion_tokens=resp.usage.completion_tokens)
return resp.choices[0].message.content
@retry(
retry=retry_if_exception_type((openai.RateLimitError, openai.APITimeoutError)),
wait=wait_exponential_jitter(initial=2, max=30),
stop=stop_after_attempt(4),
before_sleep=before_sleep_log(log, structlog.stdlib.INFO),
reraise=True,
)
def _call_with_retries(prompt: str) -> str:
return _primary_call(prompt)
def call_llm_resilient(prompt: str) -> dict:
"""Returns {content, source, degraded} — never raises."""
start = time.monotonic()
try:
content = _call_with_retries(prompt)
return {"content": content, "source": "primary", "degraded": False}
except pybreaker.CircuitBreakerError:
log.warning("circuit_open_fallback", reason="circuit_breaker")
except RetryError as exc:
log.warning("retries_exhausted_fallback", error=str(exc))
except openai.APIError as exc:
log.error("api_error_fallback", status=getattr(exc, 'status_code', None), error=str(exc))
# Fallback: cheaper, faster model on a separate client (different quota pool)
try:
content = _primary_call(prompt, model="gpt-4o-mini")
latency_ms = int((time.monotonic() - start) * 1000)
log.info("fallback_success", model="gpt-4o-mini", latency_ms=latency_ms)
return {"content": content, "source": "fallback", "degraded": True}
except Exception as exc:
log.error("fallback_failed", error=str(exc))
return {"content": "", "source": "error", "degraded": True}How this code works
This code ensures reliable interactions with an LLM, even when services face issues. The call_llm_resilient function attempts to get a response from a powerful LLM and gracefully handles failures, always returning a result rather than raising an error to the caller.
It works by first wrapping primary LLM calls (_primary_call) with a pybreaker.CircuitBreaker named _breaker. This _breaker automatically "trips" after fail_max=5 consecutive failures, temporarily stopping further calls to prevent overwhelming a struggling service and giving it time to recover before reset_timeout=60 seconds. On top of this, the _call_with_retries function uses tenacity's @retry decorator. It specifically retries openai.RateLimitError and openai.APITimeoutError with an wait_exponential_jitter delay up to stop_after_attempt(4) times, logging attempts with before_sleep_log. If these mechanisms fail, such as catching a pybreaker.CircuitBreakerError or RetryError, the code falls back. A subtle but important detail is the fallback to gpt-4o-mini which uses a different, often less contended, model and potentially a separate quota pool. The function's robust design means it never raises an exception to the calling code; instead, it returns a dictionary indicating success, fallback, or error, providing degraded: True when a fallback was used.
Practice & master
Try the exercise, check your understanding, then mark this lesson mastered to track your path to pro.
Exercise
Build a resilient_embed function that calls OpenAI's embedding API with retry on rate limits, falls back to a local sentence-transformers model if retries exhaust, and returns a dict with the embedding vector plus a source field indicating which path was used. Test it by mocking the OpenAI call to always raise RateLimitError.
# sentence-transformers==2.7.0 tenacity==8.2.3 openai==1.30.0
from tenacity import retry, stop_after_attempt, wait_exponential_jitter, retry_if_exception_type, RetryError
import openai
# from sentence_transformers import SentenceTransformer
client = openai.OpenAI(api_key="sk-test") # replace with env var
# local_model = SentenceTransformer("all-MiniLM-L6-v2")
@retry(
# TODO: retry only on RateLimitError and APITimeoutError
# TODO: use exponential jitter wait, max 20s, 3 attempts
)
def _openai_embed(text: str) -> list[float]:
# TODO: call client.embeddings.create with model="text-embedding-3-small"
pass
def resilient_embed(text: str) -> dict:
# TODO: call _openai_embed; on RetryError fall back to local_model.encode(text).tolist()
# TODO: return {"embedding": [...], "source": "openai" or "local"}
pass
# Quick test
if __name__ == "__main__":
result = resilient_embed("Hello, world")
print(result["source"], len(result["embedding"]))Quick check
Your LLM workflow retries on every exception type including 401 Unauthorized. What is the most likely consequence?
A circuit breaker is in the 'half-open' state. What does the next incoming request trigger?
You store your circuit breaker state in-process (Python variable). Your service runs 10 pods. What problem does this create?