# Local LLM Benchmark Methodology [2026]: TTFT vs tok/s Done Right

> Stop screenshot-benchmarking. Here’s a reproducible local LLM benchmark methodology for 2026 that separates TTFT from throughput and reports rerunnable results.

- Canonical: https://www.kunalganglani.com/blog/local-llm-benchmark-methodology
- Author: Kunal Ganglani
- Published: 2026-08-01 · Updated: 2026-08-01
- Category: Technology · Tags: local-llm, benchmarks, llama-cpp, quantization, ai-hardware

## TL;DR

Running a model on your own laptop or GPU box is only useful if you can measure it honestly. Most “local LLM benchmarks” are just a tokens-per-second screenshot that hides the stuff that changes performance: how long the prompt is, how big the context window is, what quantization you used, and whether you tested one chat session or many at once. A good benchmark separates time to first token from steady generation speed, repeats runs, and publishes results in a format someone else can rerun. If you do that, you can finally compare machines fairly and figure out why your setup feels slow.

Local LLM benchmark methodology 2026 is the line between “my laptop gets 12 tok/s” and actually knowing whether your setup is good for interactive chat, agent tool-use, or boring background batch jobs.

And yeah, I’m going to be a little rude about it: most local LLM “benchmarks” floating around are screenshots and vibes. They hide the stuff that actually moves performance: prompt length, context length, batch size, quantization, KV cache settings, thermals, power mode, and sometimes even whether your terminal UI is the bottleneck.

Here’s the stance I’m taking: **a benchmark that can’t be reproduced is content, not data.**

**Key takeaways**

- A single tokens/sec number is useless without context length, batch size, and whether you measured prefill or decode.
- Report **Time To First Token (TTFT)** and **steady-state decode tok/s** separately. They map to different real workloads.
- Benchmark across multiple context lengths (at least 512, 2k, 8k, 32k). Long-context changes everything.
- Quantization benchmarks only mean something if you hold the model family and prompt suite constant.
- Publish a machine-readable report (JSONL) plus a short table. Include warmups, median, and p95.
> If your benchmark can’t tell me TTFT and context length, it’s not a benchmark. It’s a vibe check.

Based on the benchmark data I maintain at [kunalganglani.com/llm-benchmarks](https://www.kunalganglani.com/llm-benchmarks), the most common failure mode is people optimizing for the wrong metric.

Apple Silicon unified memory will happily *load* models that would OOM on discrete GPUs, but throughput can still disappoint. And quantization quality cliffs are model-family-specific, so a blanket “just run Q4” recommendation is wrong.

## What you’re actually measuring (TTFT, prefill, decode)

When people say “tokens per second,” they usually mean **decode throughput**. That’s the steady-state speed once the model has digested the prompt and is just cranking out the next token. It matters for long generations, batch jobs, and anything where you care about total throughput.

![Nvidia logo on a green background with abstract spheres.](https://cdn.sanity.io/images/vzekdneq/production/0a35f113b2a927c93b894c441a8e48d93a53f915-1200x675.webp)

But interactive tools live or die on **Time To First Token (TTFT)**. TTFT is mostly the **prefill** phase: processing the prompt (plus any retrieved context) and building the KV cache before the first output token can show up.

If you don’t separate these phases, your comparisons are fake:

- A model that feels “snappy” at a 512-token prompt can feel like sludge at 16k.
- A backend tuned for throughput can look great on tok/s and still be miserable for TTFT.
This is why “0.50 tok/s” anecdotes from community threads are basically noise without the missing variables. You see the same pattern in posts like the HN discussion where someone runs a huge model, reports one throughput number, and leaves out context length and quantization: [Hacker News thread](https://news.ycombinator.com/item?id=49123386).

## A reproducible local LLM benchmark harness (template)

I’m not going to dump a giant codebase here. You don’t need my scripts. You need the **schema** and the discipline.

![Two nvidia titan x graphics cards side by side](https://cdn.sanity.io/images/vzekdneq/production/6bff3660c4bf6b93bbf82600b21a04db16dd33ea-1200x675.webp)

If two people can’t run the same benchmark and get results that at least rhyme, you’re not benchmarking. You’re narrating.

### Inputs you must lock down

For each run, log:

- `model_id` (exact GGUF/GPTQ/etc identifier)
- `quantization` (e.g., `Q4_K_M`, `Q8_0`, `FP16`)
- `backend` + version/commit (e.g., `llama.cpp` commit hash)
- `device` (CPU/GPU) and key knobs (threads, GPU layers)
- `n_ctx` (context length)
- `prompt_tokens`, `output_tokens`
- sampler settings: `temperature`, `top_p`, `top_k`, `repeat_penalty`
- batching/concurrency: `batch_size`, `concurrency`
If you skip any of these, you’ll end up “discovering” performance differences you caused yourself.

### Outputs you must record

- `ttft_ms`
- `prefill_tok_s`
- `decode_tok_s`
- `total_latency_ms`
- `ram_gb`, `vram_gb` (or unified memory usage where applicable)
- `cpu_util_pct`, `gpu_util_pct`
- `power_mode` (laptop on battery vs plugged in is not the same benchmark)
I don’t care if your util numbers are imperfect. I care that you *log something* so you can explain what happened.

### JSONL report schema (copy-paste)

Use one JSON object per line. Example fields:

- `timestamp`, `host`, `os`
- `backend`, `backend_version`
- `model_id`, `quant`
- `n_ctx`, `prompt_tokens`, `output_tokens`
- `ttft_ms`, `prefill_tok_s`, `decode_tok_s`
- `batch_size`, `concurrency`
- `notes` (thermal throttling, background tasks)
This is boring. That’s why it works.

## The 2026 controls checklist (so results are comparable)

Local inference performance swings for stupid reasons. That’s why “performance videos” are a genre.

![black fan device close-up photography](https://cdn.sanity.io/images/vzekdneq/production/76beafb78b6f19b7ed6235232857078a023801c2-1200x675.webp)

For example, [Alex Ziskind](https://www.youtube.com/watch?v=L9QZ97y9Exg) has a whole piece arguing local LLMs can be “10x slower than they should be” with the wrong settings. I agree with the core point. I disagree with the vibe some people take away, which is “just copy these magic flags.”

The fix isn’t magic flags. The fix is controlling variables.

Here’s my baseline checklist:

1. **Thermals:** run after 5 minutes of steady load or force high-performance mode. Log if you hit throttling.
1. **Warmup:** do at least 2 warmup runs per configuration.
1. **Threads pinned:** set CPU threads explicitly. “Auto” is fine for personal use and terrible for comparisons.
1. **Streaming timing:** measure TTFT from first byte received, not after printing to a terminal UI.
1. **KV cache type:** log it. It changes memory bandwidth behavior.
1. **Same prompt suite:** same exact prompts, same tokenization, same output token budget.
1. **Temperature:** use `temperature=0` for deterministic runs when measuring raw performance. Then do a second pass at your real settings if you care about product realism.
Notice what’s not on the list: “trust me, bro.”

## Context length benchmarking: what to test and why

If you only test short prompts, you’re benchmarking a world that barely exists anymore.

2026 usage is long-context by default: RAG dumps, agent scratchpads, tool schemas, codebases, endless chat history because nobody ever clears the thread.

Minimum context lengths I recommend:

- **512** tokens: toy chat baseline
- **2k** tokens: realistic short conversation
- **8k** tokens: RAG with a few retrieved chunks
- **32k** tokens: repo-scale context or long tool traces
Why these numbers?

- They force you to see the curve. Prefill time is roughly proportional to prompt tokens.
- They expose memory limits and KV cache behavior.
If you’ve read my take on [RAG context windows](/blog/rag-context-window-limitations), you already know “just stuff more tokens” is usually the wrong move. Good benchmarks make that trade visible instead of pretending bigger context is a free upgrade.

## Quantization benchmarks: speed vs quality without lying to yourself

Quantization is where benchmarking goes to die.

People compare:

- different quant levels
- different model families
- different context lengths
- different backends
…and then declare “Q4 is fine.” No. You just mixed confounders until the conclusion became meaningless.

What I do instead:

- Fix one model family and one model size.
- Run **the same prompt suite** at each quant.
- Measure performance *and* a simple correctness/quality score (even if it’s crude).
The boring truth, based on what I’ve seen in my benchmark database, is that quantization quality cliffs are model-family-specific. A blanket “just run Q4” is advice for getting surprised later.

If you want the deeper taxonomy, I’ve got it in [LLM quantization](/blog/llm-quantization-levels-q4-q8-fp16) and [GGUF vs GPTQ vs EXL2](/blog/llm-quantization-gguf-gptq-exl2).

## Batching and concurrency: throughput mode vs interactive mode

Throughput and latency are enemies. Batching usually increases throughput and usually makes TTFT worse.

So you need two benchmark modes. If your “benchmark” doesn’t say which mode it’s testing, I’m not reading it.

### Interactive mode (chat, IDE assist, agent loop)

- `concurrency = 1`
- small batch size
- latency budget focus: TTFT and p95 decode latency
### Throughput mode (document processing, background jobs)

- `concurrency >= 4` (or whatever your target is)
- larger batches
- measure tokens/sec aggregated across requests
This distinction matters if you’re choosing between “run locally on my Mac” and “run on a dedicated GPU box.” I talk about the cost side in [local LLM break-even math](/blog/local-llm-cost-breakeven) and the workflow side in [run local LLMs in VS Code](/blog/run-local-llm-vscode).

## Task-based benchmarks for developers (still reproducible)

Synthetic prompts are fine for microbenching.

They’re not fine for deciding what to buy.

A developer-oriented mini-suite I like:

1. **Repo Q&A (RAG):** ask 10 fixed questions over a pinned repo snapshot. Measure TTFT at 8k and 32k.
1. **JSON extraction:** fixed input docs, strict schema output. Score by JSON validity plus field accuracy.
1. **Code patching:** apply a small diff request. Score by `git apply` success plus tests passing.
Keep it reproducible:

- pin the repo commit
- pin the input files
- cap output tokens
- use `temperature=0` for the scoring run
If you’re building [AI agents](/pillars/ai-agents) or doing [production AI](/pillars/production-ai), this maps to what your system actually does. Not what a leaderboard says you should care about.

## Comparing backends fairly (llama.cpp, vLLM, MLX, TensorRT-LLM)

Backends aren’t interchangeable. They’re optimized for different shapes, and pretending otherwise is how you end up posting a “vLLM is slow” take that’s really just “I configured it like a CLI.”

Here’s what to log and how to compare:

- **Backend identity and version:** a “llama.cpp result” without a commit hash is basically useless. Link the project: [ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp).
- **Serving vs CLI:** `vLLM` is a serving engine. Compare it in a server config, not by invoking a one-off CLI and calling it “slow.” Reference: [vllm-project/vllm](https://github.com/vllm-project/vllm).
- **Apple Silicon:** MLX is a real ecosystem now. If you benchmark Macs, say whether you used Metal via llama.cpp or MLX. Reference: [MLX](https://github.com/ml-explore/mlx).
- **NVIDIA optimization stacks:** if you’re using TensorRT-LLM, say so and describe build/runtime details. Reference: TensorRT-LLM.
Rule: **only compare systems that are configured for the same objective** (interactive vs throughput) and the same model/quant.

## Reporting format: make screenshot benchmarks obsolete

If you want your results to be useful to other people, publish two things:

1) A compact table for humans.

| Model | Quant | Backend | n_ctx | Concurrency | TTFT (ms) median | Decode tok/s median | Decode tok/s p95 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| … | … | … | 8192 | 1 |  |  |  |

2) A JSONL artifact for machines.

Also publish run statistics:

- warmups: **2**
- measured runs: **10** (minimum)
- report median + p95, not just mean
Why 10? Because local machines are noisy. Background processes, clock scaling, thermal drift. Ten runs gives you variance you can actually see instead of pretending you measured a constant.

If you care about benchmarking discipline in general, see my post on [benchmarks and p99 latency](/blog/jemalloc-vs-malloc-tcmalloc-p99-latency).

## Measurement pitfalls that will embarrass you

A few common traps I keep seeing:

- **Terminal rendering overhead:** printing tokens to a terminal can distort decode speed. Time the stream before rendering.
- **Power limits:** laptops on battery can drop performance hard. Log plugged vs unplugged.
- **Thermal throttling:** the first run is fast, the next five are slower, and you “average” them like it’s a law of nature.
- **Mismatched output tokens:** comparing a 64-token generation to a 1024-token generation will skew results.
- **KV cache reuse:** some setups reuse caches or session states. If you’re testing cold-start TTFT, clear state.
None of these are exotic. That’s the problem.

## FAQ

### How do you benchmark a local LLM reliably?

Use a fixed prompt suite, fixed model+quant, and record TTFT and decode tok/s separately. Run warmups, then 10 measured runs, and publish median and p95. Log backend version/commit and the key knobs like context length, batch size, and concurrency.

### What is time to first token (TTFT) and why does it matter?

TTFT is the time from sending a request to receiving the first generated token. It captures prompt processing (prefill) costs, which dominate in long-context and RAG workloads. If TTFT is bad, the model will feel laggy even if decode throughput is high.

### What affects tokens per second for llama.cpp/Ollama?

Quantization level, context length, batch size, CPU thread settings, and whether layers run on GPU all change tok/s. Thermal state and power mode can also swing results. If you don’t log those, you can’t explain performance differences.

### How does context length change local LLM speed?

Longer context increases prefill work and often increases memory bandwidth pressure due to a larger KV cache. That usually increases TTFT and can reduce steady-state throughput depending on hardware and backend. Benchmark at multiple context lengths to see the curve.

### How do quantization levels (e.g., 4-bit vs 8-bit) affect speed and quality?

Lower-bit quantization usually reduces memory usage and can increase speed, but it can degrade output quality in model-specific ways. The only honest way to compare is to keep prompts and scoring fixed and measure both latency and correctness. Don’t generalize from one model to all models.

### What is the difference between throughput and latency in LLM inference?

Latency is how long a single request takes (TTFT and per-token delay). Throughput is how much total work the system can do per unit time, often improved by batching and concurrency. Interactive chat cares more about latency. Batch processing cares more about throughput.

## What comes next

In 2026, “local AI” buying decisions are increasingly about workflow shape, not peak tokens/sec.

People are running longer contexts, more concurrent sessions, and more tool-use JSON than ever. If you keep posting single-number screenshots, you’ll keep making bad hardware and backend decisions.

My challenge: next time you share a local LLM benchmark, publish TTFT and decode tok/s at 2k and 32k, plus a JSONL artifact. If you can’t, don’t call it a benchmark.

Photo by Bernd 📷 Dittrich on Unsplash.

## FAQ

### How do you benchmark a local LLM reliably?

Lock down the model, quantization, prompts, and sampler settings. Measure TTFT and decode tokens/sec separately, run warmups, then at least 10 measured runs, and publish median and p95 along with backend version and context length.

### What is time to first token (TTFT) and why does it matter?

TTFT is the time from sending a request to receiving the first generated token. It mostly reflects prompt processing cost, which gets much worse with long context and RAG. Bad TTFT makes a model feel slow even if decode throughput is high.

### What batch size should I use when benchmarking LLMs?

Use batch size based on your workload. For interactive chat, keep batching low and concurrency at 1 to reflect latency. For throughput jobs, increase batching and concurrency, but report them explicitly because they change TTFT and tokens/sec.

### How many runs should I average for stable results?

Do 2 warmup runs to stabilize caches and clocks, then 10 measured runs minimum. Report median and p95 rather than only the mean, because local machines have variance from thermals and background load.

### Should I benchmark with temperature 0?

Yes for baseline performance and reproducibility, because it reduces randomness in generation. If you care about real product behavior, also run a second pass at your typical temperature and report both.