HardwareModelSearch

82 of 82 rows

Hardware ↑	Model	Quant	Tokens/sec	Runtime	Source	Note
M1 · 16 GB	Llama 3.2 3B	`Q4_K_M`	30	`ollama`	● measured
M1 · 16 GB	Llama 3.1 8B	`Q4_K_M`	12	`ollama`	● measured
M1 Max · 32 GB	Llama 3.1 8B	`Q4_K_M`	42	`ollama`	● measured
M1 Max · 64 GB	Llama 3.1 70B	`Q4_K_M`	4.5	`ollama`	● measured	Tight fit at 64 GB
M1 Pro · 16 GB	Llama 3.1 8B	`Q4_K_M`	22	`ollama`	● measured
M1 Pro · 32 GB	Llama 3.1 8B	`Q5_K_M`	19	`ollama`	● measured
M1 Ultra · 128 GB	Llama 3.1 70B	`Q4_K_M`	8.5	`ollama`	● measured
M2 · 16 GB	Llama 3.1 8B	`Q4_K_M`	18	`ollama`	● measured
M2 Max · 32 GB	Llama 3.1 8B	`Q4_K_M`	45	`ollama`	● measured
M2 Max · 64 GB	Qwen 2.5 32B	`Q4_K_M`	11	`ollama`	● measured
M2 Max · 96 GB	Llama 3.1 70B	`Q4_K_M`	8	`ollama`	● measured
M2 Pro · 16 GB	Llama 3.1 8B	`Q4_K_M`	24	`ollama`	● measured
M2 Pro · 32 GB	Qwen 2.5 14B	`Q4_K_M`	14	`ollama`	● measured
M2 Ultra · 192 GB	Llama 3.1 70B	`Q5_K_M`	13	`ollama`	● measured
M2 Ultra · 192 GB	Mixtral 8x22B (MoE)	`Q4_K_M`	18	`mlx`	● measured
M3 · 16 GB	Llama 3.1 8B	`Q4_K_M`	20	`ollama`	● measured
M3 · 24 GB	Qwen 2.5 14B	`Q4_K_M`	7	`ollama`	● measured
M3 Max · 36 GB	Llama 3.1 8B	`Q4_K_M`	46	`ollama`	● measured
M3 Max · 36 GB	Qwen 2.5 14B	`Q4_K_M`	28	`ollama`	● measured
M3 Max · 64 GB	Qwen 2.5 32B	`Q4_K_M`	14	`ollama`	● measured
M3 Max · 64 GB	Codestral 22B	`Q4_K_M`	20	`ollama`	● measured
M3 Max · 128 GB	Llama 3.1 70B	`Q4_K_M`	9	`ollama`	● measured
M3 Pro · 18 GB	Llama 3.1 8B	`Q4_K_M`	22	`ollama`	● measured
M3 Pro · 36 GB	Qwen 2.5 14B	`Q4_K_M`	11	`ollama`	● measured
M3 Ultra · 256 GB	Llama 3.1 70B	`Q8_0`	11	`mlx`	● measured
M3 Ultra · 512 GB	DeepSeek-V3 671B (MoE)	`Q4_K_M`	17	`mlx`	● measured	The headline benchmark for M3 Ultra 512 GB
M3 Ultra · 512 GB	Llama 3.1 405B	`Q4_K_M`	4	`mlx`	● measured
M4 · 16 GB	Llama 3.1 8B	`Q4_K_M`	23	`ollama`	● measured
M4 · 24 GB	Qwen 2.5 14B	`Q4_K_M`	12	`ollama`	● measured
M4 Max · 36 GB	Llama 3.1 8B	`Q4_K_M`	58	`mlx`	● measured
M4 Max · 48 GB	Qwen 2.5 32B	`Q4_K_M`	22	`ollama`	● measured
M4 Max · 64 GB	Qwen 2.5 32B	`Q5_K_M`	19	`ollama`	● measured
M4 Max · 64 GB	Qwen 2.5 Coder 32B	`Q4_K_M`	22	`ollama`	● measured
M4 Max · 128 GB	Llama 3.1 70B	`Q4_K_M`	12	`mlx`	● measured
M4 Max · 128 GB	Llama 3.3 70B	`Q4_K_M`	11	`mlx`	● measured
M4 Max · 128 GB	Qwen 2.5 72B	`Q4_K_M`	11	`mlx`	● measured
M4 Pro · 24 GB	Llama 3.1 8B	`Q4_K_M`	34	`ollama`	● measured
M4 Pro · 48 GB	Qwen 2.5 32B	`Q4_K_M`	11	`ollama`	● measured
M4 Pro · 48 GB	Qwen 2.5 Coder 32B	`Q4_K_M`	11	`ollama`	● measured
M4 Pro · 64 GB	Llama 3.1 70B	`Q4_K_M`	5	`ollama`	● measured	Slow but usable for batch tasks
M5 · 24 GB	Llama 3.1 8B	`Q4_K_M`	30	`ollama`	○ reference	Estimated from M5 bandwidth uplift over M4
M5 Max · 64 GB	Qwen 2.5 32B	`Q4_K_M`	24	`mlx`	○ reference
M5 Max · 128 GB	Llama 3.3 70B	`Q4_K_M`	13	`mlx`	○ reference
M5 Pro · 48 GB	Qwen 2.5 32B	`Q4_K_M`	13	`ollama`	○ reference
RX 7800 XT	Llama 3.1 8B	`Q4_K_M`	68	`llama.cpp`	● measured
RX 7900 XTX	Llama 3.1 8B	`Q4_K_M`	105	`llama.cpp`	● measured	ROCm — works but driver setup is fragile
RX 7900 XTX	Qwen 2.5 14B	`Q4_K_M`	62	`llama.cpp`	● measured
RX 7900 XTX	Qwen 2.5 32B	`Q4_K_M`	30	`llama.cpp`	● measured
RTX 3080 10GB	Llama 3.1 8B	`Q4_K_M`	70	`llama.cpp`	● measured
RTX 3060 12GB	Llama 3.1 8B	`Q4_K_M`	38	`llama.cpp`	● measured
RTX 3060 12GB	Qwen 2.5 14B	`Q4_K_M`	18	`llama.cpp`	● measured
RTX 4070	Llama 3.1 8B	`Q4_K_M`	78	`llama.cpp`	● measured
RTX 4060 Ti 16GB	Llama 3.1 8B	`Q4_K_M`	50	`llama.cpp`	● measured
RTX 4060 Ti 16GB	Qwen 2.5 14B	`Q4_K_M`	26	`llama.cpp`	● measured
RTX 4070 Ti Super	Llama 3.1 8B	`Q4_K_M`	95	`llama.cpp`	● measured
RTX 4070 Ti Super	Qwen 2.5 14B	`Q4_K_M`	54	`llama.cpp`	● measured
RTX 4080	Llama 3.1 8B	`Q4_K_M`	100	`llama.cpp`	● measured
RTX 4080	Qwen 2.5 14B	`Q4_K_M`	60	`llama.cpp`	● measured
RTX 5070 Ti	Llama 3.1 8B	`Q4_K_M`	125	`llama.cpp`	○ reference
RTX 5080	Llama 3.1 8B	`Q4_K_M`	140	`llama.cpp`	○ reference
RTX 5080	Qwen 2.5 14B	`Q4_K_M`	88	`llama.cpp`	○ reference
RTX 3090	Llama 3.1 8B	`Q4_K_M`	92	`llama.cpp`	● measured
RTX 3090	Qwen 2.5 14B	`Q4_K_M`	55	`llama.cpp`	● measured
RTX 3090	Qwen 2.5 32B	`Q4_K_M`	28	`llama.cpp`	● measured
RTX 3090	Codestral 22B	`Q5_K_M`	32	`llama.cpp`	● measured
RTX 4090	Llama 3.1 8B	`Q4_K_M`	145	`llama.cpp`	● measured
RTX 4090	Qwen 2.5 14B	`Q4_K_M`	92	`llama.cpp`	● measured
RTX 4090	Qwen 2.5 32B	`Q4_K_M`	42	`llama.cpp`	● measured
RTX 4090	Qwen 2.5 Coder 32B	`Q4_K_M`	42	`llama.cpp`	● measured
RTX 4090	Codestral 22B	`Q5_K_M`	48	`llama.cpp`	● measured
RTX 4090	Mixtral 8x7B (MoE)	`Q4_K_M`	50	`llama.cpp`	● measured	Just fits in 24 GB at Q4
RTX 5090	Llama 3.1 8B	`Q4_K_M`	220	`llama.cpp`	● measured
RTX 5090	Qwen 2.5 14B	`Q4_K_M`	145	`llama.cpp`	● measured
RTX 5090	Qwen 2.5 32B	`Q4_K_M`	75	`llama.cpp`	● measured
RTX 5090	Qwen 2.5 Coder 32B	`Q5_K_M`	68	`llama.cpp`	● measured
RTX 5090	Mixtral 8x7B (MoE)	`Q5_K_M`	76	`llama.cpp`	● measured
A100 40GB	Llama 3.1 8B	`fp16`	95	`vllm`	● measured
RTX 6000 Ada	Llama 3.1 70B	`Q5_K_M`	22	`vllm`	● measured
RTX A6000	Llama 3.1 70B	`Q4_K_M`	18	`llama.cpp`	● measured	Single A6000 holds 70B at Q4 with overhead
A100 80GB	Llama 3.1 70B	`fp16`	28	`vllm`	● measured
H100 80GB	Llama 3.1 70B	`fp16`	55	`vllm`	● measured
H100 80GB	Llama 3.3 70B	`fp16`	55	`vllm`	● measured

ChipUnified RAMModelQuantTokens/sec (sustained)RuntimeSource URL (optional — helps us verify)Note (optional)Your email (optional — only if you want credit)

About this dataset

Each row records measured (or carefully-sourced) sustained generation rate (tokens/sec) for a specific hardware + model + quantization combination. Sources include r/LocalLLaMA community benchmarks, Ollama GitHub discussions, MLX reference numbers, llama.cpp PR threads, and published hardware reviews.

● measured rows are direct community reports. ○ reference rows are extrapolated from bandwidth + scaling (mostly newer chips where measurements are still scarce).

Data is licensed CC BY 4.0 — use it, embed it, cite back to https://www.kunalganglani.com/llm-benchmarks and we're square.

Have measurements to share? Use the submission form below — reviewed within 48h.

Local LLM Benchmark Database

Submit a benchmark

About this dataset