GGUF vs GPTQ vs EXL2: LLM Quantization Compared [2026]

LLM quantization is the process of reducing a large language model's weight precision — from 16-bit floating point down to 8-bit, 4-bit, or even lower — to shrink its memory footprint and run it on consumer hardware. Three formats dominate this space in 2026: GGUF, GPTQ, and EXL2. This guide compares all three head-to-head on quality loss, speed, VRAM, and tooling so you can pick the right one for your hardware.

Key takeaways:

• GGUF is the universal format — it runs on CPU, Apple Silicon, NVIDIA, and AMD via llama.cpp, and it's the only format supported by Ollama and LM Studio.
• GPTQ delivers slightly better quality per bit on NVIDIA GPUs because it uses calibration data to minimize per-row reconstruction error, but it's CUDA-only.
• EXL2 is the speed king on NVIDIA — ExLlamaV2 delivers the fastest token generation on consumer GPUs, and its non-integer bit-widths (2.5-bit, 3.5-bit) let you fine-tune the quality/VRAM trade-off with surgical precision.
• Q4_K_M is the sweet spot for most users — roughly 95% of fp16 quality at about 40% of the VRAM cost, based on the benchmark data I maintain at kunalganglani.com/llm-benchmarks.
• Hugging Face acquired ggml.ai in 2026, making GGUF an officially supported first-class format on the Hub and consolidating the two biggest open-source LLM ecosystems.

What Is LLM Quantization and Why Does It Matter?

A 7B parameter model in fp16 eats approximately 14GB of memory. A 70B model? Around 140GB. Unless you're running a multi-GPU datacenter rig, you're not loading that into VRAM without quantization.

Quantization maps high-precision floating-point weights to lower-precision integer representations. The trade-off is straightforward: fewer bits per weight means less memory and faster inference, but some information is lost. The question is how much you lose and whether it matters for your use case.

As Tim Dettmers, co-author of the QLoRA paper, demonstrated: a 65B model can be fine-tuned on a single 48GB GPU at 4-bit precision while preserving full 16-bit task performance. That finding established the baseline understanding that 4-bit quantization can retain near-fp16 quality — but the devil is in the details of which 4-bit method you use.

Q4_K_M hits roughly 95% of fp16 quality at 40% of the VRAM cost. For most local LLM tasks, that's the only number that matters.

The real complexity isn't whether to quantize. It's choosing among 20+ quantization methods — Hugging Face's Transformers documentation now lists GGUF, GPTQ, AWQ, bitsandbytes, EXL2, HIGGS, HQQ, BitNet, AQLM, and more. That fragmentation is the problem this guide solves. I'm narrowing the field to the three formats that actually matter for local LLM deployment: GGUF, GPTQ, and EXL2.

The Three Main Formats: GGUF, GPTQ, and EXL2 at a Glance

Before diving into each format, here's the comparison table that should anchor your decision:

The short version: if you're on Apple Silicon or want maximum compatibility, GGUF. If you're in the Hugging Face Transformers ecosystem on NVIDIA, GPTQ. If you want the absolute fastest token generation on an NVIDIA GPU and don't mind a smaller ecosystem, EXL2.

GGUF Deep Dive: How llama.cpp Quantization Works

GGUF (GPT-Generated Unified Format) is the file format created by Georgi Gerganov for llama.cpp, the open-source C/C++ inference engine with 119,000 GitHub stars — more than any other LLM inference project. GGUF replaced the older GGML binary format to be more extensible, centralizing all model metadata (special tokens, RoPE scaling parameters, vocabulary) in a single self-contained file.

As Maxime Labonne, Machine Learning Researcher, explains: a 7B model has approximately 35 transformer layers, and llama.cpp's GPU offload feature lets you push a subset of those layers to VRAM while keeping the rest on CPU. This hybrid CPU+GPU inference is GGUF's killer feature. No other format supports it.

This matters because most people running local AI don't have 24GB of VRAM. They have a MacBook with unified memory, or a desktop with a 12GB card. GGUF lets them run models that technically exceed their VRAM by spilling layers to system RAM. It's slower than full GPU offload, but it works.

GGUF Quantization Tiers: Q4_0, Q4_K_M, Q5_K_M, Q8_0

Not all 4-bit quantization is equal. The "K" in Q4_K_M stands for k-quants, a method that uses mixed precision across different parts of the model. Understanding the difference between Q4_0 and Q4_K_M is critical:

• Q4_0: Uniform 4-bit quantization. Every weight gets the same treatment. This is the oldest and lowest-quality 4-bit option.
• Q4_K_M: Mixed 4-bit and 6-bit precision. Attention layers and embeddings — the weights that matter most for quality — get 6-bit precision while less sensitive layers stay at 4-bit. The result is dramatically lower perplexity degradation than Q4_0 at only a modest VRAM increase.
• Q5_K_M: The middle ground. Uses 5-bit and 6-bit mixed precision. Roughly 10-15% more VRAM than Q4_K_M, with measurably less quality loss.
• Q8_0: 8-bit quantization. Virtually indistinguishable from fp16 on perplexity benchmarks. Uses about twice the memory of Q4_K_M.

The theoretical foundation here comes from the QLoRA research. LLM weights follow an approximately normal distribution, which is why NF4 (4-bit NormalFloat) — which bins quantization levels to match the normal curve — outperforms uniform int4. GGUF's k-quants apply a similar principle: allocate more precision to high-importance weights, less to the rest.

What Is Imatrix Quantization?

Imatrix (importance matrix) quantization is a calibration technique in llama.cpp that significantly improves Q4 quality. Instead of treating all weights equally, it runs a calibration dataset through the model to identify which weights contribute most to output quality. Those weights get quantized more carefully.

From maintaining the benchmark database on this site, I've found that quantization quality cliffs are model-family-specific — a blanket Q4 recommendation is wrong. Some model architectures tolerate aggressive quantization gracefully, while others fall apart below Q5. Imatrix calibration helps close that gap, particularly on models that are more sensitive to quantization. If you're downloading a Q4_K_M GGUF, check whether the uploader used imatrix calibration. It makes a real difference.

GPTQ Deep Dive: Per-Row Weight Quantization for CUDA GPUs

GPTQ is a post-training quantization algorithm, not a file format per se. It quantizes each row of the weight matrix independently to find a compressed version that minimizes reconstruction error. As the Hugging Face GPTQ documentation explains, weights are stored as int4 but restored to fp16 on the fly during inference.

This is an important distinction. GPTQ is a storage compression format — the actual compute still runs in fp16. That means GPTQ models need CUDA GPUs. There's no CPU fallback, no Apple Silicon support, no ROCm path (though limited AMD support exists through some third-party efforts).

Where GPTQ shines is quality-per-bit. Because it uses calibration data during quantization and minimizes error per weight row (not per block like basic GGUF quants), GPTQ at 4-bit can actually outperform GGUF Q4_K_M on perplexity benchmarks. The gap is small — typically 0.05-0.15 perplexity points on a 7B model — but it's consistent.

The implementation is handled by the GPT-QModel Python package (formerly AutoGPTQ). It integrates directly into Hugging Face Transformers, which means you can load a GPTQ model with the same API you'd use for an fp16 model. For teams already building on the HF ecosystem, this is the path of least resistance.

The trade-off: GPTQ locks you into NVIDIA CUDA hardware. If you ever want to run your model on a Mac, on CPU, or on AMD — you need a different format.

EXL2 Deep Dive: Non-Integer Bit Widths and ExLlamaV2

EXL2 is the quantization format native to ExLlamaV2, created by the pseudonymous developer turboderp. ExLlamaV2 describes itself as "a fast inference library for running LLMs locally on modern consumer-class GPUs," and it delivers on that promise.

EXL2's differentiator is non-integer bit widths. While GPTQ gives you 4-bit or 8-bit (integers only), EXL2 supports 2.5-bit, 3.0-bit, 3.5-bit, 4.0-bit, 4.5-bit, 5.0-bit, and everything in between. This isn't just a gimmick. It means you can quantize a model to exactly 3.5 bits per weight — fitting a model that's too big for your VRAM at 4-bit but would be too degraded at 3-bit. It's per-layer bit allocation, giving you surgical control over the quality/VRAM trade-off.

With 4.6K GitHub stars and 338 forks, ExLlamaV2 has a much smaller ecosystem than llama.cpp. But among NVIDIA GPU enthusiasts who care about maximizing tokens-per-second on a single consumer card, it's the tool of choice. On an RTX 4090 with 24GB VRAM, EXL2 typically generates 15-30% more tokens per second than GPTQ or GGUF at the same bit-width and model size.

The caveat: EXL2 is NVIDIA-only, and its ecosystem is small enough that finding pre-quantized models can be harder than GGUF or GPTQ. You may need to quantize models yourself.

Head-to-Head: Quality Loss at Each Quantization Tier

This is where most guides fail. They describe the formats but never show you what you actually lose. Here's what the numbers look like, using a 7B-class model as the baseline:

A few things jump out from this data. Q8_0 is essentially lossless. The perplexity increase is within measurement noise. Q4_K_M is where you start seeing real but tolerable degradation. And below 3.5 bits, quality falls off a cliff — especially for code generation, where small errors in reasoning compound into broken logic.

If you want to reproduce these quality-loss numbers (or compare them fairly across GGUF, GPTQ, and EXL2), I lay out the exact testing setup in Local LLM Benchmark Methodology [2026]: TTFT vs tok/s Done Right, including how I separate time-to-first-token from steady-state throughput.

Coding accuracy degrades faster than general text quality at lower bit-widths. I've observed this repeatedly when running benchmarks for this site's local LLM hardware guides. A model that still produces coherent prose at Q4_K_M might generate subtly incorrect function signatures or miss edge cases in code. If your primary use case is AI coding, stay at Q5_K_M or higher.

Head-to-Head: Speed (Tokens per Second) on the Same Hardware

Speed varies dramatically by format, hardware, and whether the model fits entirely in VRAM. Here's the hierarchy on an NVIDIA RTX 4090 (24GB) with a model fully loaded into VRAM:

The story changes completely on Apple Silicon. On an M4 Max with 128GB unified memory, GGUF through llama.cpp or Ollama is your only option — and it's surprisingly fast. Unified memory changes the VRAM-is-the-limit intuition. Big models load just fine on Apple Silicon, but throughput is the real trade-off versus a discrete NVIDIA GPU.

For vLLM production serving, GPTQ and AWQ are the supported formats. EXL2 and GGUF aren't options there.

VRAM Requirements by Format and Model Size

This table is the single most practical reference for deciding what you can actually run:

The practical upshot: a 24GB GPU (RTX 3090, 4090, or RTX 5090) can run a 7B model at fp16, a 13B at Q8_0, a 34B at Q4_K_M, or squeeze a 70B at EXL2 3.5-bit if you accept the quality trade-off. These numbers exclude the memory needed for KV cache during inference, so plan for 1-3GB of overhead depending on context length.

For the best quantization format for a 24GB GPU, the answer depends on model size. Running a 7B? Use Q8_0 — you have the room. Running a 13B or 33-34B? Q4_K_M or Q5_K_M is the sweet spot. Trying to run a 70B? EXL2 at 3.0-3.5 bits is your only option for full GPU offload, and you should expect noticeable quality loss.

Which LLM Quantization Format Should You Use? Decision Framework

Stop overthinking this. Here's the decision tree:

You're on a Mac or want CPU inference → GGUF. It's the only format with real CPU and Apple Silicon support. Use Q4_K_M for most tasks, Q5_K_M or Q8_0 if you have unified memory to spare.

You have an NVIDIA GPU and want maximum speed → EXL2 via ExLlamaV2. Pick a bit-width that fits your VRAM. The non-integer precision lets you extract every MB of value from your card.

You're building with Hugging Face Transformers or need [production AI](/pillars/llm-hardware-local-ai) serving → GPTQ. It integrates natively with the HF ecosystem and is supported by vLLM for production inference.

You have an AMD GPU → GGUF via llama.cpp with ROCm. AMD support in GPTQ and EXL2 ranges from limited to nonexistent.

You want to use [Ollama](/blog/ollama-vs-lm-studio) or [LM Studio](/blog/lm-studio-vs-jan) → GGUF. These tools use llama.cpp under the hood and only support GGUF models.

You want the highest quality at 4-bit → GPTQ slightly edges GGUF Q4_K_M because of its per-row calibrated error minimization. EXL2 at 4.0 bits with calibration is also excellent. The differences are small enough that tooling convenience should be the deciding factor.

Tooling Compatibility: Ollama, LM Studio, text-generation-webui, vLLM

This is the table that doesn't exist anywhere else. Every inference front-end has format preferences:

If you only use one tool, your format choice is already made. If you switch between Ollama for development and vLLM for production, you'll likely maintain two different quantized versions of the same model.

2026 Update: ggml.ai Acquired by Hugging Face — What Changes?

The biggest structural change to the local LLM ecosystem in 2026 is Hugging Face's acquisition of ggml.ai — the company founded by Georgi Gerganov in 2023 that created llama.cpp and the GGUF format. Nat Friedman and Daniel Gross provided the original pre-seed funding; now Hugging Face owns the whole operation.

What this means practically:

GGUF becomes a first-class Hugging Face format. The Hub already supported GGUF uploads, but post-acquisition, expect native GGUF model cards with quantization metadata, integrated perplexity scores, and one-click quantization from fp16 to any GGUF tier directly on the Hub.

Ecosystem convergence. Hugging Face Transformers already added GGUF loading support. With ggml.ai in-house, the integration between llama.cpp's inference engine and HF's model distribution will tighten. The days of managing separate GGUF download workflows are numbered.

Long-term support guarantee. Before the acquisition, GGUF depended on one developer and a small team. Now it has Hugging Face's resources behind it. For anyone building AI in production on GGUF, this is the single strongest signal that the format has staying power.

The TheBloke gap is closing. Tom Jobbins (TheBloke) published 3,863+ quantized model repos on Hugging Face before stopping in January 2024. That left a distribution vacuum. In 2026, model publishers increasingly ship their own GGUF variants, automated quantization pipelines on the Hub fill remaining gaps, and the HF/ggml.ai integration will likely make "publish GGUF alongside fp16" a default checkbox for model uploaders.

Every comparison guide published in 2023 or 2024 misses this entirely. The competitive dynamics between GGUF and the HF-native formats (GPTQ, AWQ, bitsandbytes) have fundamentally shifted now that GGUF is owned by Hugging Face. The format war is consolidating, not fragmenting.

When Is GPTQ Actually Better Than GGUF?

GPTQ and GGUF Q4_K_M both target ~4 bits per weight, but they get there differently. GPTQ quantizes each weight matrix row independently using a calibration dataset, minimizing the reconstruction error for each row. GGUF k-quants use block-level quantization with mixed precision by layer type.

The result: GPTQ often produces lower perplexity than GGUF Q4_K_M at the same effective bit-width. The gap is small (typically 0.05-0.15 perplexity points on a 7B model), but it's real and consistent across model families.

So when should you actually pick GPTQ over GGUF?

1. You're exclusively on NVIDIA CUDA and don't need CPU or Mac support
2. You're integrating with Hugging Face Transformers or LangChain and want the simplest loading path
3. You need serving via vLLM for production workloads
4. You're pushing quality-per-bit to the limit and every 0.1 perplexity point matters

If none of those apply, GGUF's portability advantage outweighs GPTQ's slight quality edge. This is one of those things where the boring answer is actually the right one.

Conclusion and Recommendations

The LLM quantization landscape has consolidated significantly in 2026. Hugging Face now owns the most important inference format (GGUF) and the most important model distribution platform. GPTQ remains the CUDA workhorse for the Transformers ecosystem. EXL2 is the enthusiast's choice for raw speed on NVIDIA consumer cards.

My recommendation for most developers: start with GGUF Q4_K_M. It runs everywhere — Mac, Linux, Windows, CPU, GPU — through Ollama or llama.cpp. If you're hitting quality issues on specific tasks, step up to Q5_K_M or Q8_0 before switching formats. If you're chasing maximum throughput on an RTX 4090 or 5090, give EXL2 a try — the speed difference is real.

The format you pick today is less risky than it was two years ago. With Hugging Face consolidating GGUF into its platform, the conversion and interoperability story will only get better. The real question for 2027 isn't which format wins — it's whether quantization below 4 bits gets good enough to make the 70B tier accessible on a single consumer GPU without meaningful quality loss. I'm betting it does.

Photo by Patrick Hendry on Unsplash.