# How to Run Qwen 35B on 16GB VRAM [2026]: Flags + Quants

> A reproducible 16GB recipe for “Qwen 35B”: which Qwen2.5-32B quants fit, how to budget KV cache, and the exact serving flags that stop OOMs.

- Canonical: https://www.kunalganglani.com/blog/run-qwen-35b-16gb-vram
- Author: Kunal Ganglani
- Published: 2026-09-16 · Updated: 2026-09-16
- Category: AI and Machine Learning · Tags: local-llm, qwen, quantization, llama-cpp, vllm

## TL;DR

You can run “Qwen 35B” class models on a 16GB GPU in 2026, but only if you treat memory like a budget. The weights are just the start. The hidden cost is KV cache, which grows with your context length and causes most out-of-memory crashes after the model loads. The practical recipe is: use Qwen2.5-32B-Instruct, start with a 4-bit quant, cap context at 4k, and only increase once you’ve measured peak VRAM and speed. If you need multi-user serving, switch to vLLM and limit how much VRAM it can grab.

If you want **how to run Qwen 35B on 16GB VRAM** in 2026, here’s the outcome: you can get a “35B-class” Qwen model running on a consumer 16GB card without CPU-offload. The prerequisite that trips people up is **KV cache**. If you don’t budget it, your model loads, you generate 50 tokens, and then you hit an OOM spike and blame the runtime.

This post is a recipe. You’ll get:

- which “Qwen 35B” people actually mean in 2026
- quantization picks that fit 16GB with room for runtime overhead
- KV-cache math + a context-length table you can use before you download anything
- working flag sets for `llama.cpp` and `vLLM` that avoid the usual fragmentation/OOM traps
- a tiny benchmark harness that logs **tokens/sec** and **peak VRAM**
I’m opinionated about this: **16GB is enough for a 30B–40B dense model if you treat VRAM like a budget, not a vibe.**

## What is “Qwen 35B” in 2026?

“Qwen 35B” is a search shorthand for running a **Qwen-family large language model** in the ~30B–40B parameter class (the model size people want for “close to flagship” quality) on consumer GPUs.

![a pair of black and silver graphics cards](https://cdn.sanity.io/images/vzekdneq/production/6fb8ebd9d958d3ba3e5701dfcd7a7bf6be95290d-1200x675.webp)

In practice, most “Qwen 35B” queries map to **Qwen2.5-32B-Instruct** (32B parameters), because it’s a common sweet spot: big enough to feel meaningfully better than 7B/14B, small enough to squeeze onto 16GB with aggressive quantization.

If you’re picking a variant today:

- Prefer **instruction-tuned** for chat and coding prompts: `Qwen2.5-32B-Instruct` on Hugging Face.
- Prefer **base** only if you’re doing your own fine-tuning or controlled evals.
Here’s the canonical reference for the instruct model: [Qwen/Qwen2.5-32B-Instruct](https://huggingface.co/Qwen/Qwen2.5-32B-Instruct).

Two adjacent realities worth saying out loud:

1. The “35B” number is not magic. Your quality comes from the specific model family, training recipe, and your quantization, not the integer in the name.
1. On 16GB, **context length is a first-class tradeoff**, not a checkbox. 16k context is possible, but it changes everything.
If you want broader model-selection context, I keep a running set of picks in [local LLM](/pillars/llm-hardware-local-ai) guides and the more opinionated list in [best local model for agentic coding](/blog/best-local-model-agentic-coding).

## Which quantizations fit in 16GB (and which ones are a trap)

Most guides say “use 4-bit” and leave it there. That’s how you end up with a model that technically runs but is unstable at your target context.

![Two computer graphics cards on a yellow background](https://cdn.sanity.io/images/vzekdneq/production/5d15ccf93a993837c9e2ecd44834c1a4cbc0f5b2-1200x675.webp)

### Weight memory: the boring math

A 32B model has ~32 billion parameters.

- FP16 weights: `32B * 2 bytes ≈ 64 GB` (not happening on a 16GB card)
- 8-bit weights: `32B * 1 byte ≈ 32 GB`
- 4-bit weights: `32B * 0.5 bytes ≈ 16 GB`
That last line is why people fixate on 4-bit. But **16GB for weights alone leaves you zero headroom** for KV cache, CUDA workspace, kernels, and allocator fragmentation. In the real world, you want weights to land closer to **12–14 GB** if you want the rest of the system to breathe.

That’s why my default recommendations for 16GB are:

- **GGUF (llama.cpp): Q4_K_M** for “it fits and doesn’t feel awful”
- **GGUF (llama.cpp): Q5_K_M** only if you accept smaller context (or partial GPU offload)
- **GPTQ 4-bit / AWQ 4-bit (vLLM/exllama-ish stacks):** good for throughput, but pick conservative max length
If you want a deeper primer on formats, I already wrote [GGUF vs GPTQ vs EXL2: LLM quantization compared](/blog/llm-quantization-gguf-gptq-exl2) and [LLM quantization levels compared: Q4_K_M vs Q8_0 vs FP16](/blog/llm-quantization-levels-q4-q8-fp16).

### Quality expectations (no mythology)

4-bit quantization quality loss is real, but it’s not uniform.

- For many model families, Q4 is “surprisingly fine” on instruction-following.
- For others, Q4 turns into brittle reasoning and weird refusals.
Based on the benchmark data I maintain at **kunalganglani.com/llm-benchmarks**, the pattern I keep seeing across model families is that **a blanket “Q4 is always fine” recommendation is wrong**. There are quality cliffs, and they’re model-specific. (This is exactly why I’m obsessed with reproducible harnesses, which you’ll get later in this post.)

If you’re optimizing for coding help and you can tolerate slightly lower context, I’d rather run **a better quant (Q5-ish) at 4k–8k** than a worse quant at 16k. Tokens you can’t use effectively are wasted tokens.

## How to compute VRAM usage (weights + KV cache + runtime)

If you only take one thing from this post, take this: **your OOMs are usually KV cache, not weights.**

![White graphics card with three fans and rtx logo](https://cdn.sanity.io/images/vzekdneq/production/ed00146c3fbd725219f30a1f7c8a3b972908e1b9-1200x675.webp)

### What is KV cache?

KV cache (key-value cache) is the memory the model uses to store attention “history” for each generated token, so it doesn’t have to recompute attention over the full prompt every time.

KV cache grows roughly linearly with:

- context length (prompt + generated tokens)
- number of layers
- hidden size
- number of KV heads (or grouped-query attention setup)
- bytes per element (FP16 vs FP8-ish vs quantized cache, depending on runtime)
### The safe budgeting model I use

On a 16GB GPU, I budget like this:

- **12–14 GB** for weights (depending on quant)
- **1–3 GB** for KV cache (depends on context)
- **1–2 GB** for runtime overhead (CUDA graphs, workspaces, fragmentation, driver overhead)
That last line is why “4-bit equals 16GB” is a trap. You need slack.

### Practical VRAM budget table (16GB target)

These are conservative “don’t pagefault, don’t explode” targets. The exact KV cache size depends on the runtime and cache dtype, but the trend is stable.

| Context length | KV cache budget (target) | Notes on 16GB |
| --- | --- | --- |
| 4k | ~1–2 GB | Usually stable even with heavier runtimes |
| 8k | ~2–4 GB | Starts colliding with fragmentation and graphs |
| 16k | ~4–8 GB | Only if weights are well under 12–13 GB and runtime is tuned |

When people say “it loads but OOMs when generating”, they’re typically trying **8k–16k** on a setup that had no slack.

If you’re doing long-context Retrieval-Augmented Generation (RAG), read [RAG context window limits](/blog/rag-context-window-limitations). Bigger context is often the wrong lever anyway.

## Pick a serving stack for 16GB (and the flags that actually prevent OOM)

I’ll be blunt: if your goal is “get Qwen 35B-class running on 16GB with minimum suffering,” start with **`llama.cpp` GGUF**. Then graduate to `vLLM` if you need multi-user serving and higher throughput.

I’ve shipped enough systems around deterministic gates in my own pipeline (the one that publishes this blog) to trust one principle: **simple, observable systems beat clever systems when you’re chasing tail latencies and weird failure modes**. The same applies to local LLM serving.

### Option A: llama.cpp (GGUF) for single-user or small-team

`llama.cpp` is the most reliable “it just runs” stack for constrained VRAM.

Official repo: [llama.cpp](https://github.com/ggerganov/llama.cpp) by [Georgi Gerganov](https://github.com/ggerganov).

Flags that matter on 16GB:

- `--ctx-size` / `-c`: your context window. This is the KV cache lever.
- `--n-gpu-layers`: how many layers to offload to GPU. Use this to fit when you’re barely over budget.
- `--batch-size` (or prompt processing batch): too high increases peak memory.
- `--flash-attn` (if available in your build): can improve speed, but can change memory behavior.
A starting command pattern (edit paths, and start at 4096 context):

```bash
./llama-server \
  -m /models/qwen2.5-32b-instruct-q4_k_m.gguf \
  -c 4096 \
  --n-gpu-layers 999 \
  --threads 8 \
  --port 8080
```

If you OOM, don’t randomly change 10 flags. Do this sequence:

1. Drop `-c` from **8192 → 4096** (largest impact, most predictable)
1. If you still OOM, reduce GPU offload: `--n-gpu-layers 999 → 60 → 40`
1. If you still OOM, choose a smaller quant (Q4 over Q5)
I care about reproducibility here because I’ve seen “fixes” that only work until a different prompt shape hits production. Your goal is not “it ran once.”

If you want a GUI wrapper, see my take on [Ollama vs LM Studio 2026](/blog/ollama-vs-lm-studio-2026) and [Ollama vs llama.cpp 2026](/blog/ollama-vs-llama-cpp).

### Option B: vLLM (AWQ/GPTQ) for throughput and multi-user serving

If you want to serve multiple users, you want `vLLM`. It’s a real serving engine with scheduling, batching, and memory efficiency.

Official docs: [vLLM](https://docs.vllm.ai/en/latest/) and the GitHub repo [vLLM](https://github.com/vllm-project/vllm) by vLLM contributors.

The flag that matters most on 16GB is the one that stops `vLLM` from greedily allocating all memory:

- `--gpu-memory-utilization` (set a cap, like `0.85` or `0.90`)
A pragmatic server command pattern:

```bash
python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-32B-Instruct \
  --dtype half \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.88 \
  --port 8000
```

Concrete numbers that matter:

- `0.88` means “use at most **88%** of VRAM”, leaving ~**12%** for driver/runtime overhead.
- `--max-model-len 4096` is not conservative for fun. It’s because 16GB + long context is where you’ll see allocator fragmentation and sudden spikes.
If you need a broader serving decision framework, I wrote [how to serve a local LLM to multiple users](/blog/serve-local-llm-multiple-users).

### Option C: ExLlamaV2 / EXL2 (when you want max speed per VRAM)

EXL2-style quantization can be extremely fast for single-GPU inference. The tradeoff is operational complexity and “it depends” behavior across kernels and driver setups.

If you’re already in the exllama ecosystem, your main lever is still the same: **context length drives cache**. Cap it early, then increase.

I’m not linking a specific command here because exllama packaging shifts faster than most blog posts get updated, and “copy-pasteable but wrong” is worse than no snippet.

If you want the format comparison, see [GGUF vs GPTQ vs EXL2](/blog/llm-quantization-gguf-gptq-exl2).

## What breaks on consumer GPUs (and how to debug OOMs without guessing)

The annoying part about 16GB is not “it’s small.” It’s that you’re operating near the cliff, so **tiny overheads matter**.

Here are the failures I see most often:

1. **Windows WDDM overhead**: you have 16GB on paper but less in practice. A 0.5–1.5 GB hit is normal depending on what else is running.
1. **Allocator fragmentation**: you have enough free VRAM total, but not enough contiguous space for a big allocation.
1. **CUDA graph spikes**: graphs can increase memory usage for speed. Great when you have headroom. Pain when you don’t.
1. **Flash-attn / attention kernel mismatch**: kernels that improve throughput can change workspace needs.
1. **Driver/toolkit mismatch**: you “fixed” it by reinstalling, but you didn’t learn what broke.
My debugging loop is boring on purpose:

- Log **peak VRAM** for every run.
- Change exactly one variable at a time: quant, context, or offload.
- Keep a “known good” baseline prompt to compare.
If your workload is agentic and tool-heavy, you’ll also want to think about **LLM cost** and request shaping even when you run locally. Local doesn’t mean free. It means you pay in watts and latency. See [local LLM break-even math](/blog/local-llm-break-even-cost-model) and [LLM cost](/blog/agent-per-task-cost-calculation).

## A tiny benchmark harness: tokens/sec + peak VRAM (markdown output)

Most benchmarks are not benchmarks. They’re screenshots.

Here’s a minimal harness approach that works across stacks:

- fixed prompt
- fixed max generated tokens (e.g. **256**)
- capture tokens/sec from the runtime output
- capture peak VRAM from `nvidia-smi`
- emit a markdown table so you can paste results into notes or issues
### 1) Fixed prompt file

Create `prompt.txt` with a stable prompt. Keep it consistent across runs.

Example prompt (short enough to fit in 4k even with system tokens):

- ask for a design doc
- ask for edge cases
- ask for a small code sample
### 2) Run + log peak VRAM

A shell harness pattern:

```bash
#!/usr/bin/env bash
set -euo pipefail

MODEL="$1"      # path or HF id
CTX="$2"        # e.g. 4096
OUT="$3"        # output label

# Sample peak VRAM every 200ms during the run
( while true; do nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits; sleep 0.2; done ) \
  | awk '{ if ($1>max) max=$1 } END { print max }' \
  > "${OUT}.peak_vram_mb" &
SMI_PID=$!

START=$(date +%s)

# Example: llama.cpp CLI (adjust to your binary)
./llama-cli -m "$MODEL" -c "$CTX" -n 256 -f prompt.txt \
  2>&1 | tee "${OUT}.log"

END=$(date +%s)
kill $SMI_PID || true

echo "seconds=$((END-START))" > "${OUT}.time"
```

### 3) Produce a markdown table row

Parse out:

- tokens/sec from the runtime log
- peak_vram_mb from the sampled file
Then you can build a table like:

| Runtime | Quant | Context | tok/s | Peak VRAM (MB) |
| --- | --- | --- | --- | --- |
| llama.cpp | Q4_K_M | 4096 | 18.7 | 15240 |
| llama.cpp | Q5_K_M | 4096 | 16.1 | 15980 |

The point is not perfect scientific rigor. The point is to stop guessing.

If you want to do benchmarking “for real”, read [local LLM benchmark methodology](/blog/local-llm-benchmark-methodology) and [LLM latency benchmark methodology](/blog/llm-latency-benchmark-methodology).

## My default 16GB recipe (the one I’d tell a teammate to use)

If you’re on an RTX 4080 16GB or any similar 16GB GPU, and your goal is “Qwen 35B-class, stable, useful”:

1. Use **Qwen2.5-32B-Instruct**.
1. Start with **GGUF Q4_K_M** in `llama.cpp`.
1. Set `--ctx-size 4096`. Earn 8192 with real logs.
1. If you need serving, move to `vLLM` and cap `--gpu-memory-utilization` to **0.85–0.90**.
1. Benchmark every change. Don’t trust vibes.
One more thing. The local AI crowd keeps trying to brute-force context length as if it’s free. It’s not. My prediction: in 2026, the winners won’t be the people running 32B at 16k context on a 16GB GPU. It’ll be the people who run 32B at 4k–8k with tight retrieval, good prompts, and a benchmark harness that stops regressions from shipping.

Photo by GAMERCOMP.RU on Unsplash.