The core mental model is that a large language model is a stateless function: tokens in, tokens out. A serving framework like vLLM wraps that function in an HTTP server, adds continuous batching (so multiple requests share the GPU at the same time), manages the KV cache across requests, and exposes an OpenAI-compatible REST endpoint. You point your existing code at http://your-server:8000/v1/chat/completions and swap the model name. The API shape is nearly identical to OpenAI's, so migration is often a config change.
The GPU choice drives everything downstream. A 7B parameter model like Mistral-7B-Instruct fits on a single A10G (24GB VRAM) at bfloat16 and can serve roughly 500-1000 tokens/second with continuous batching, depending on sequence lengths. A 70B model needs 2-4 A100-80GB cards. Quantization with GPTQ or AWQ cuts memory roughly in half at a small quality penalty, letting you fit a 13B model on a 24GB card or a 70B on two 40GB cards. Cloud GPU costs vary by provider and region, but as an illustration: an A10G spot instance might run around $0.60-1.20/hr depending on cloud and market conditions. At that rate, sustained throughput of 1M tokens/hr brings the per-token cost well below $0.001, compared to $0.002-0.015 per 1K tokens on flagship managed APIs. Do your own math with current pricing.
A real-world scenario: a B2B SaaS company uses an LLM to generate first-draft email replies for customer support agents. They process roughly 500,000 emails per month. At GPT-4o rates (illustrative), that's a meaningful monthly bill. They evaluate Llama-3-8B-Instruct and find it handles 85% of cases well enough for agents to lightly edit. They deploy two A10G instances behind a load balancer using vLLM. The monthly instance cost (on-demand, not spot) is a fraction of the API bill, and they keep GPT-4o calls as a fallback for the complex 15%. Total cost drops significantly. The tradeoff: they now own the uptime problem, model versioning, and monitoring.
The main alternative approaches are: (1) smaller managed API models like gpt-4o-mini or Claude Haiku, which cost much less than flagships and require zero ops; (2) serverless GPU platforms like Modal, RunPod Serverless, or Replicate, which give you open-source models without cluster management but charge per-second with cold start latency; (3) quantized models via GGUF + llama.cpp on CPU-only servers for very low-throughput tasks where latency is forgiving. Each fits a different point on the ops-complexity vs cost-control curve. The sibling lesson on model selection covers the managed-API tradeoffs; this lesson focuses on the self-hosted route.
Scale changes the problem in specific ways. At 10 users you are probably over-provisioned and paying for idle GPU time. At 10K users you are hitting real sustained load, and the economics start to favor self-hosting for the right tasks. At 10M users you are dealing with multi-region deployment, rolling model updates without downtime, autoscaling GPU node pools, and SLA requirements. At that scale you likely run a mix: self-hosted for your high-volume bread-and-butter tasks, managed APIs for edge cases requiring frontier model capability. Kubernetes with GPU node pools (GKE Autopilot with A100s, or EKS with p4d instances) lets you autoscale deployments. vLLM's distributed serving mode with tensor parallelism spans a single model across multiple GPUs or nodes.
Reliability and operational overhead are the real costs people undercount. You need: health checks and readiness probes on the serving pods, alerting on GPU utilization and error rates, a process for loading new model weights without a cold restart (vLLM supports live model swapping in newer versions), and a circuit-breaker that routes to your managed-API fallback when latency spikes. Structured logging of every request with token counts lets you track utilization and debug performance regressions. Model updates require a staging deploy, eval comparison against your regression test suite (covered in the evaluation subtopics), and a rollout strategy. That's real engineering work. Budget it honestly before committing.
Key Takeaways
- Use vLLM or TGI as your serving layer; they handle batching, KV-cache, and multi-GPU automatically.
- Calculate your break-even point: fixed GPU cost per hour divided by tokens-per-hour at target utilization.
- Self-hosting makes sense when GPU utilization stays above ~40% across your billing period.
- Keep a managed API fallback in your code for when your self-hosted cluster is down or overloaded.
Pro tips
- Run continuous batching benchmarks with realistic concurrency before committing to a GPU SKU. A single A10G with vLLM and a batch size of 32 concurrent requests often outperforms naive single-request benchmarks by 8-10x in tokens/second, which completely changes your cost math.
- Use tensor parallelism (
--tensor-parallel-size 2) across two GPUs rather than pipeline parallelism for latency-sensitive workloads. Pipeline parallelism maximizes throughput but adds inter-GPU bubble latency that hurts P99. - Set
--max-model-lenin vLLM to the longest context you actually need, not the model's theoretical maximum. KV cache scales with context length, so capping it lets you fit more concurrent requests in VRAM. - Track your GPU utilization as a billing signal. If sustained utilization drops below 35-40%, you are likely paying for idle capacity. Move to a serverless GPU platform (Modal, RunPod) for that workload, or consolidate models onto fewer instances.
Common pitfalls
- Mistake: Benchmarking throughput with a single request and assuming it scales linearly. Fix: Always benchmark with concurrent load matching your p95 production concurrency. vLLM's continuous batching only shows its value under real concurrency.
- Mistake: Storing model weights in the container image, causing 20-40GB image pulls on every pod restart. Fix: Mount weights from a shared network volume (EFS, GCS FUSE) or use an init container to pull weights once to a node-local cache.
- Mistake: Skipping the fallback path because self-hosted feels reliable. Fix: GPU hardware fails, nodes get preempted, OOM errors happen. Always wire in a managed API fallback with a circuit breaker before going to production.
- Mistake: Running the same heavyweight model for all tasks to simplify ops. Fix: Deploy a small quantized model (3B-7B) for classification or extraction tasks and a larger one only for generation. Separate deployments cut cost and improve utilization on each.
When to self-host vs use managed APIs vs serverless GPU
| Option | Use when | Avoid when |
|---|---|---|
| Self-hosted (vLLM / TGI on dedicated GPUs) | Sustained high throughput (>1M tokens/day), strict data residency, or predictable load where GPU utilization stays above ~40%. | Spiky or low-volume traffic, small team with no MLOps capacity, or when you need frontier-model quality that open-source cannot match. |
| Managed API (OpenAI, Anthropic, Google) | Low to medium volume, experimental/early-stage products, need for frontier-model capability, or zero desire to manage infrastructure. | Strict data residency requirements, or when per-token costs are consuming a large share of gross margin at scale. |
| Serverless GPU (Modal, RunPod Serverless, Replicate) | Medium volume with spiky patterns, want open-source models without cluster management, or prototyping before committing to dedicated hardware. | Latency-critical paths where cold-start (10-30s) is unacceptable, or sustained load where per-second billing exceeds dedicated GPU cost. |
| CPU inference (llama.cpp / GGUF quantized) | Very low throughput, latency-tolerant batch jobs, or edge/on-premise deployments with no GPU access. | Any workload requiring more than ~20-50 tokens/second per instance, or models larger than 13B even with aggressive quantization. |
Code Example
# vLLM >= 0.4.0 must be installed: pip install vllm
# Start the server: python -m vllm.entrypoints.openai.api_server \
# --model mistralai/Mistral-7B-Instruct-v0.3 --port 8000
from openai import OpenAI
# Point the OpenAI client at your local vLLM server
client = OpenAI(
api_key="not-needed", # vLLM ignores this but the client requires it
base_url="http://localhost:8000/v1",
)
response = client.chat.completions.create(
model="mistralai/Mistral-7B-Instruct-v0.3",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Summarize the key risk factors in three bullets."},
],
max_tokens=256,
temperature=0.2,
)
print(response.choices[0].message.content)
print(f"Tokens used: {response.usage.total_tokens}")How this code works
This code demonstrates how to interact with a self-hosted open-source large language model (LLM) using the familiar OpenAI client library, a powerful strategy for cost optimization and high-volume AI tasks. After ensuring vllm is installed via pip install vllm and starting the local api_server with a chosen model like Mistral-7B-Instruct-v0.3 on port 8000, the Python script connects to it. The from openai import OpenAI line imports the necessary client. An OpenAI() client instance is then created, crucially setting its base_url to http://localhost:8000/v1 to direct requests to the local server. A subtle point here is api_key="not-needed"; the OpenAI client library expects an API key, but the local vLLM server ignores it, so this placeholder satisfies the client.
Once configured, the client.chat.completions.create() method sends a prompt to the self-hosted LLM. The model parameter specifies which local model to use, matching the one launched by the api_server. The messages list provides the conversation history, structured with role and content for system and user instructions. Parameters like max_tokens limit the output length, and temperature controls the creativity of the response. Finally, the code prints the generated response.choices[0].message.content and response.usage.total_tokens, showing the model's reply and the tokens consumed. This entire process allows developers to leverage powerful open-source models efficiently for high-volume, cost-sensitive AI tasks.
Production-grade example
Streaming, retries with backoff, managed-API fallback, env-var auth, and structured cost logging.
# vLLM >= 0.4.0 | openai >= 1.10.0 | tenacity >= 8.0
import logging
import os
import time
from typing import Iterator
import httpx
from openai import APIConnectionError, APIStatusError, OpenAI, RateLimitError
from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_exponential
logger = logging.getLogger(__name__)
SELF_HOSTED_URL = os.environ["VLLM_BASE_URL"] # e.g. http://10.0.1.5:8000/v1
FALLBACK_API_KEY = os.environ["OPENAI_API_KEY"]
MODEL_SELF_HOSTED = "mistralai/Mistral-7B-Instruct-v0.3"
MODEL_FALLBACK = "gpt-4o-mini"
TIMEOUT_SECONDS = 30
self_hosted_client = OpenAI(
api_key="not-needed",
base_url=SELF_HOSTED_URL,
timeout=httpx.Timeout(TIMEOUT_SECONDS),
)
fallback_client = OpenAI(api_key=FALLBACK_API_KEY)
@retry(
retry=retry_if_exception_type((APIConnectionError, RateLimitError)),
wait=wait_exponential(multiplier=1, min=1, max=16),
stop=stop_after_attempt(3),
)
def _call_self_hosted(messages: list[dict], max_tokens: int) -> Iterator[str]:
stream = self_hosted_client.chat.completions.create(
model=MODEL_SELF_HOSTED,
messages=messages,
max_tokens=max_tokens,
temperature=0.2,
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
yield delta
def generate_with_fallback(
messages: list[dict],
max_tokens: int = 512,
request_id: str = "",
) -> str:
t0 = time.monotonic()
backend = "self_hosted"
output_tokens = 0
full_response = []
try:
for token in _call_self_hosted(messages, max_tokens):
full_response.append(token)
output_tokens += 1
except (APIConnectionError, APIStatusError, Exception) as exc:
logger.warning(
"self_hosted_failure",
extra={"request_id": request_id, "error": str(exc)},
)
backend = "fallback"
resp = fallback_client.chat.completions.create(
model=MODEL_FALLBACK,
messages=messages,
max_tokens=max_tokens,
temperature=0.2,
)
full_response = [resp.choices[0].message.content]
output_tokens = resp.usage.completion_tokens
latency_ms = int((time.monotonic() - t0) * 1000)
logger.info(
"inference_complete",
extra={
"request_id": request_id,
"backend": backend,
"output_tokens": output_tokens,
"latency_ms": latency_ms,
},
)
return "".join(full_response)How this code works
This code helps optimize costs for high-volume AI tasks by prioritizing a cheaper, self-hosted language model and falling back to a more expensive commercial model only if the primary option encounters problems. This ensures both cost-efficiency and high reliability, critical for production systems.
The setup creates two OpenAI client instances: self_hosted_client connects to a local vLLM endpoint defined by VLLM_BASE_URL, while fallback_client uses OPENAI_API_KEY for a service like GPT-4o-mini. A subtle but important detail is api_key="not-needed" for the self-hosted client; while many local vLLM deployments don't need a real key, the openai library still expects the argument. The _call_self_hosted function attempts to stream responses from the local model, using the @retry decorator to automatically re-attempt calls on temporary network or rate limit errors. If the self-hosted attempt (even after retries) fails with any exception, the generate_with_fallback function seamlessly catches it and switches to the fallback_client to fulfill the request. Crucial logger.info and logger.warning calls provide visibility into which backend processed the request and its latency_ms.
Practice & master
Try the exercise, check your understanding, then mark this lesson mastered to track your path to pro.
Exercise
Start a vLLM server locally (or via Docker) with a small open-source model. Write a Python script that sends 10 concurrent requests using asyncio, measures total wall-clock time and tokens-per-second, and prints a summary. Then add a fallback that detects a connection error and routes to the OpenAI API instead.
# pip install vllm openai asyncio
# Start vLLM first:
# docker run --gpus all -p 8000:8000 vllm/vllm-openai \
# --model facebook/opt-125m (small CPU-friendly model for testing)
import asyncio
import time
from openai import AsyncOpenAI, APIConnectionError
SELF_HOSTED_URL = "http://localhost:8000/v1"
MODEL = "facebook/opt-125m"
PROMPT = "List three benefits of renewable energy."
NUM_REQUESTS = 10
self_hosted = AsyncOpenAI(api_key="x", base_url=SELF_HOSTED_URL)
async def single_request(i: int) -> dict:
# TODO: call self_hosted.chat.completions.create with PROMPT
# TODO: on APIConnectionError, fall back to openai with gpt-4o-mini
# TODO: return {"index": i, "tokens": ..., "backend": "self_hosted" or "fallback"}
pass
async def main():
t0 = time.monotonic()
# TODO: run NUM_REQUESTS concurrent calls using asyncio.gather
results = []
elapsed = time.monotonic() - t0
total_tokens = sum(r["tokens"] for r in results)
# TODO: print summary: total tokens, elapsed time, tokens/sec, backend breakdown
asyncio.run(main())Quick check
Your self-hosted vLLM instance handles 200 req/min during business hours but sits idle overnight. Which alternative is most cost-effective?
Which vLLM feature most directly reduces cost per request when you have many concurrent users?
You self-host a 7B model and your GPU utilization averages 20%. What does this most likely indicate?