Quantization
5 posts tagged #quantization
5 posts on Quantization, all in the Local LLM Hardware pillar — most read: “LLM Quantization Levels Compared: Q4_K_M vs Q8_0 vs FP16” and “LM Studio Power User Setup: Profiles, Quants, LAN Serving”.
Developer Tools LM Studio Power User Setup: Profiles, Quants, LAN Serving
My opinionated LM Studio setup: reproducible model profiles, quant hygiene rules, scripted loads via lms + /api/v1, and a secure multi-user LAN server.
AI and Machine Learning How to Run Qwen 35B on 16GB VRAM: Flags + Quants
A reproducible 16GB recipe for “Qwen 35B”: which Qwen2.5-32B quants fit, how to budget KV cache, and the exact serving flags that stop OOMs.
AI and Machine Learning Gemma 4 26B on CPU: 5 tok/s Benchmark and Production Math
A $300 Xeon from 2013 runs Gemma 4 26B at 5 tok/s with no GPU. Here's the memory bandwidth math, quantization tradeoffs, and a production decision framework.
AI and Machine Learning LLM Quantization Levels Compared: Q4_K_M vs Q8_0 vs FP16
The practitioner's guide to choosing between Q4_K_M, Q5_K_S, Q8_0, and FP16 quantization for local LLMs — with real perplexity numbers, throughput benchmarks, and per-use-case recommendations.
AI and Machine Learning GGUF vs GPTQ vs EXL2: LLM Quantization Formats Compared
A head-to-head comparison of GGUF, GPTQ, and EXL2 quantization formats with real quality, speed, and VRAM trade-offs — updated for the 2026 Hugging Face acquisition of ggml.ai.
Keep exploring
Close to this topic, not on this page.