How caching works for LLM APIs
The naive cache key is a SHA-256 hash of the concatenated model name and full prompt string. This gives you exact-match caching: the same prompt, byte-for-byte, returns the cached response instantly. This is more useful than it sounds. In production, a significant fraction of LLM traffic is repeated: the same FAQ question rephrased identically, a dashboard widget that regenerates the same summary on every page load, or a nightly batch job that processes the same seed data. Exact-match caching on Redis with a 1-hour TTL can eliminate 20-40% of upstream calls in these patterns with zero visible quality tradeoff. For a more sophisticated setup, semantic caching uses an embedding model to find cached responses for prompts that are close in meaning but not identical. You embed the incoming prompt, run an ANN query against cached embeddings in something like pgvector or Qdrant, and if the nearest neighbor is above a cosine similarity threshold (typically 0.95+), return the cached answer. This is more expensive to implement and introduces a correctness risk if the threshold is too low, so treat semantic caching as an optimization layer on top of exact-match, not a replacement for it.
Rate limiting that maps to reality
Most rate limiting tutorials show a simple token bucket per IP. That works for general APIs, but for AI APIs your real constraint is upstream: OpenAI's TPM (tokens per minute) and RPM (requests per minute) limits per API key. If you have a tier-2 key with 90,000 TPM and 3,500 RPM on gpt-4o, those are the numbers you must design around, not arbitrary per-user limits you invent. A practical architecture has two layers. The outer layer rate-limits your clients by user or API key, using Redis with a sliding window or fixed window counter. The inner layer is a token bucket that tracks your own upstream quota consumption in real time. When the inner bucket is near-empty, you start queuing requests rather than forwarding them immediately. This prevents the thundering herd problem where 50 concurrent client requests all succeed your outer rate limit but together blow your upstream quota, triggering 429s from the provider that you then have to retry with backoff. Always include a Retry-After header in your 429 responses. Clients that check this header can implement correct backoff automatically. Without it, clients typically retry on a fixed interval that makes congestion worse, not better.
Graceful degradation as a policy, not an afterthought
Graceful degradation means having an explicit, documented policy for what your service does when its dependencies are degraded. The word "graceful" is doing real work here: it means the degradation is intentional, predictable, and communicated to the caller. A useful mental model is the fallback chain. For a chat completion endpoint, a reasonable chain looks like: (1) primary model (gpt-4o), (2) cheaper/faster fallback model (gpt-4o-mini), (3) a stale cached response if one exists for a similar prompt, (4) a static canned response that at least tells the user the service is degraded. You encode this chain in code, not in a runbook. The circuit breaker pattern from distributed systems applies directly here: track error rates per upstream model, and when the error rate crosses a threshold (say, 5 errors in 10 seconds), open the circuit and route directly to the fallback without even attempting the primary. The circuit stays open for a configurable period, then enters half-open state to probe recovery. Libraries like tenacity for retries and pybreaker for the circuit breaker pattern give you these primitives in Python without building from scratch.
Tradeoffs and alternatives
The main alternative to application-level caching is provider-level prompt caching. OpenAI, Anthropic, and Google all offer some form of prompt caching where repeated system prompt prefixes are cached on their infrastructure and billed at a reduced rate. This is cheaper to implement (zero code) and handles cases where your request is identical except for a small appended user turn. The tradeoff is that you give up control: TTLs are short (typically minutes to an hour), and you cannot inspect or invalidate the cache. Use provider-level caching for long system prompts and application-level caching for complete identical requests. They are complementary. For rate limiting, an alternative to Redis is an in-process rate limiter if you are running a single-server deployment. The risk is that it breaks the moment you scale to multiple instances. Prefer Redis from day one for any production deployment. For graceful degradation, some teams use a feature flag system (LaunchDarkly, etc.) to manually switch between fallback tiers during incidents. This is a valid operational pattern but requires human intervention; the circuit breaker is automatic.
What changes at scale
At 10 users you probably do not need any of this. At 10,000 users, caching becomes essential for cost control, rate limiting becomes essential for fairness, and graceful degradation becomes essential because provider incidents will happen. At 10 million users, the architecture changes substantially. A single Redis instance becomes a Redis cluster. Your rate limiter needs to be distributed and account for multiple geographic regions. Your fallback chain may span multiple providers (OpenAI as primary, Anthropic as secondary) rather than just different model tiers within one provider. You will also need observability at every layer: cache hit rate as a metric, rate limiter reject rate per user tier as a metric, fallback activation count as an alert. Without these, you are flying blind when something goes wrong at 3am.
Key Takeaways
- Cache LLM responses at the exact-match and semantic levels to cut provider costs and latency.
- Mirror your upstream provider quota structure in your own rate limiter to avoid cascading failures.
- Design explicit fallback chains: primary model, cheaper model, cached default, static response.
- Return Retry-After headers on 429s so clients back off automatically without custom logic.
Pro tips
- Cache keys should include the model name, not just the prompt. Switching from gpt-4o to gpt-4o-mini invalidates all cached answers, and you do not want stale gpt-4o-mini responses served under gpt-4o traffic after a rollback.
- Set your rate limit window to match the provider's window. OpenAI's TPM limit resets on a per-minute boundary. If your window is 60 seconds but offset by 30 seconds, you will allow bursts that still trigger upstream 429s.
- Log cache hit rate and model fallback rate as metrics, not just as log lines. When a provider has an incident, your fallback rate metric spikes visibly within seconds. That is often faster than their own status page.
- Semantic caching is a latency optimization, not a correctness feature. Never use a similarity threshold below 0.97 for factual or task-oriented prompts. A prompt asking for Paris's population and one asking for Lyon's population can be very close in embedding space.
Common pitfalls
- Mistake: Caching responses for prompts that include user-specific data (account IDs, PII). Fix: Always strip or exclude dynamic personalized fields from the cache key, or skip caching entirely for those request types.
- Mistake: Implementing rate limiting only in-process when running multiple API server instances. Fix: Use Redis or another shared store for rate limit counters from day one, even in single-instance deployments.
- Mistake: Swallowing all exceptions in the fallback chain and returning the static response without logging the root cause. Fix: Log the specific exception type and model name before each fallback step so incidents are debuggable.
- Mistake: Setting an infinite or very long cache TTL for LLM responses without considering prompt-sensitive content. Fix: Use domain-appropriate TTLs: long (24h+) for static FAQ answers, short (5-15 min) for anything referencing current events or dynamic data.
When to use exact-match vs semantic vs provider-level caching
| Option | Use when | Avoid when |
|---|---|---|
| Exact-match cache (Redis SHA-256 key) | You expect repeated identical prompts: batch jobs, dashboard widgets, FAQ bots. | Prompts are generated dynamically and rarely repeat verbatim. |
| Semantic cache (embedding + ANN lookup) | Users ask the same question with minor phrasing variation; latency reduction matters more than implementation cost. | Prompts are factually sensitive and a 0.96 similarity score could map to meaningfully different questions. |
| Provider-level prompt caching (OpenAI, Anthropic) | You have a long static system prompt reused across many requests; zero implementation overhead is appealing. | You need cache observability, control over TTLs, or full-response caching across complete identical requests. |
| No caching | Every request is unique by design (e.g., code generation with unique user context) and correctness requires a fresh model call. | You have any repeating prompt patterns or cost pressure from high request volume. |
Code Example
# redis-py 5.x, openai 1.x
import hashlib, json, os
import redis
from openai import OpenAI
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
cache = redis.Redis(host="localhost", port=6379, decode_responses=True)
CACHE_TTL = 3600 # seconds
def cached_completion(prompt: str, model: str = "gpt-4o-mini") -> str:
cache_key = hashlib.sha256(f"{model}:{prompt}".encode()).hexdigest()
cached = cache.get(cache_key)
if cached:
return json.loads(cached)
response = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
)
result = response.choices[0].message.content
cache.setex(cache_key, CACHE_TTL, json.dumps(result))
return resultHow this code works
This code provides a smart way to get AI completions from OpenAI: it uses caching to avoid asking the AI the same question multiple times. This is essential for backend systems to save money, respond faster, and stay within API rate limits. It begins by setting up connections to the OpenAI API client and a redis.Redis cache server, with CACHE_TTL defining how long data stays in the cache before expiring, preventing stale information.
The core logic resides in the cached_completion function. When called, it first generates a unique cache_key from the input prompt and model. It then checks cache.get(cache_key) to see if a previous answer exists. If cached data is found, it's immediately returned after json.loads to convert it back from string format, saving an OpenAI API call. A subtle point here is the model parameter's default value of gpt-4o-mini: if not specified, the function silently uses this specific, often cheaper and faster model, which can affect performance and cost. If no cache hit occurs, the code calls client.chat.completions.create to fetch a new response. The result is then stored in Redis using cache.setex() with an expiration before being returned.
Production-grade example
Adds rate limiting, two-model fallback chain, retry backoff, token logging, and structured observability.
# redis-py 5.x, openai 1.x, tenacity 8.x, structlog 24.x
import hashlib, json, os, time
from contextlib import contextmanager
from typing import Optional
import redis
import structlog
from openai import OpenAI, APIStatusError, APITimeoutError
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
log = structlog.get_logger()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"], timeout=20.0)
cache = redis.Redis(host=os.environ.get("REDIS_HOST", "localhost"), port=6379, decode_responses=True)
PRIMARY_MODEL = "gpt-4o"
FALLBACK_MODEL = "gpt-4o-mini"
CACHE_TTL = 3600
RATE_LIMIT_WINDOW = 60 # seconds
RATE_LIMIT_MAX = 20 # requests per user per window
def check_rate_limit(user_id: str) -> tuple[bool, int]:
key = f"rl:{user_id}:{int(time.time()) // RATE_LIMIT_WINDOW}"
pipe = cache.pipeline()
pipe.incr(key)
pipe.expire(key, RATE_LIMIT_WINDOW)
count, _ = pipe.execute()
allowed = count <= RATE_LIMIT_MAX
retry_after = RATE_LIMIT_WINDOW - (int(time.time()) % RATE_LIMIT_WINDOW)
return allowed, retry_after
@retry(
retry=retry_if_exception_type((APITimeoutError, APIStatusError)),
wait=wait_exponential(multiplier=1, min=1, max=8),
stop=stop_after_attempt(3),
reraise=True,
)
def _call_model(prompt: str, model: str) -> tuple[str, int]:
start = time.monotonic()
resp = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
timeout=20.0,
)
latency_ms = int((time.monotonic() - start) * 1000)
tokens = resp.usage.total_tokens if resp.usage else 0
log.info("llm_call", model=model, tokens=tokens, latency_ms=latency_ms)
return resp.choices[0].message.content, tokens
def completion_with_fallback(
prompt: str,
user_id: str,
fallback_response: Optional[str] = "Service is temporarily degraded. Please try again shortly.",
) -> dict:
allowed, retry_after = check_rate_limit(user_id)
if not allowed:
log.warning("rate_limited", user_id=user_id, retry_after=retry_after)
return {"error": "rate_limited", "retry_after": retry_after, "text": None}
cache_key = hashlib.sha256(f"{PRIMARY_MODEL}:{prompt}".encode()).hexdigest()
cached = cache.get(cache_key)
if cached:
log.info("cache_hit", user_id=user_id, key=cache_key[:8])
return {"text": json.loads(cached), "cached": True, "model": PRIMARY_MODEL}
for model in (PRIMARY_MODEL, FALLBACK_MODEL):
try:
text, tokens = _call_model(prompt, model)
cache.setex(cache_key, CACHE_TTL, json.dumps(text))
return {"text": text, "cached": False, "model": model, "tokens": tokens}
except (APITimeoutError, APIStatusError) as exc:
log.error("model_failed", model=model, error=str(exc))
log.error("all_models_failed", user_id=user_id)
return {"text": fallback_response, "cached": False, "model": "static", "degraded": True}How this code works
This code provides a robust system for interacting with AI models, focusing on caching, rate limiting, and graceful degradation to enhance reliability and efficiency. It aims to reduce unnecessary API calls, manage user access, and maintain service availability even when external AI services encounter issues. The central completion_with_fallback function orchestrates these behaviors. It first uses check_rate_limit with redis INCR and EXPIRE to track requests per user, enforcing RATE_LIMIT_MAX within a RATE_LIMIT_WINDOW. If permitted, it checks for a cache_key derived from the prompt in redis, logging a cache_hit and returning cached data immediately if available.
If no cached response exists, the system attempts to call the PRIMARY_MODEL via _call_model. This function is decorated with @retry from tenacity, automatically reattempting calls that fail due to APITimeoutError or APIStatusError with an wait_exponential backoff. Should the PRIMARY_MODEL fail even after retries, the system gracefully degrades by attempting the FALLBACK_MODEL. A subtle aspect is that while a cache_key is always generated based on the PRIMARY_MODEL, a successful response from the FALLBACK_MODEL will be cached under this PRIMARY_MODEL key. Consequently, a future cache_hit will report PRIMARY_MODEL as the source, even if the content originated from the FALLBACK_MODEL. If all model calls ultimately fail, a static fallback_response is returned, preventing a complete service disruption.
Practice & master
Try the exercise, check your understanding, then mark this lesson mastered to track your path to pro.
Exercise
Build a FastAPI endpoint POST /complete that accepts a JSON body with fields prompt (string) and user_id (string). The endpoint must: (1) enforce a rate limit of 5 requests per user per 60 seconds using Redis, returning 429 with a Retry-After header when exceeded; (2) check an exact-match Redis cache before calling the OpenAI API; (3) cache successful responses with a 10-minute TTL.
# fastapi 0.111+, redis-py 5.x, openai 1.x
import hashlib, json, os, time
from fastapi import FastAPI, HTTPException, Response
from pydantic import BaseModel
import redis
from openai import OpenAI
app = FastAPI()
cache = redis.Redis(host="localhost", port=6379, decode_responses=True)
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
RATE_LIMIT_MAX = 5
RATE_LIMIT_WINDOW = 60
CACHE_TTL = 600
class CompletionRequest(BaseModel):
prompt: str
user_id: str
@app.post("/complete")
def complete(req: CompletionRequest, response: Response):
# TODO 1: Check rate limit for req.user_id
# Return 429 with Retry-After header if exceeded
# TODO 2: Build cache key from prompt, check Redis
# Return cached value if present
# TODO 3: Call OpenAI, store result in cache, return result
passQuick check
Your upstream LLM provider returns a 429. Your circuit breaker is open. Which response should your API send to the client?
A Redis rate limit counter is incremented per user per 60-second window. Why is this approach risky on multi-instance deployments if you use an in-process counter instead?
When would exact-match caching fail to provide any benefit even for a FAQ-style chatbot?