Quantisation Explained: GGUF, GPTQ and Small VRAM

The first time I tried to run a 13B model on a card with 12GB of VRAM, I hit an out-of-memory error before the model had finished loading. The maths is unforgiving: a 13-billion-parameter model in the FP16 precision it was likely trained and released in needs roughly 26GB just to hold the weights, before you’ve allocated a single byte for the context window. My card had less than half that. Quantisation is the technique that made it possible anyway, and understanding what it actually does — rather than treating it as a magic “make it smaller” checkbox — is the difference between picking a sensible trade-off and quietly running a model that’s dumber than it needs to be.

Why quantisation exists

A model’s weights are just numbers, and the precision you store those numbers in determines both the memory footprint and, to a point, the quality of what the model produces. Full precision is typically FP16 or BF16 — 16 bits per weight. Quantisation reduces that to fewer bits: 8, 4, sometimes lower, by mapping the original range of floating-point values onto a smaller set of discrete levels, then storing a scale factor per group of weights so the compressed values can be reconstructed close to their original magnitude at inference time.

The core trade-off is exactly what it sounds like: fewer bits per weight means less memory and (usually) faster inference, in exchange for some accumulated error in every weight’s value. The interesting part — and the reason quantisation works as well as it does — is that transformer weights are surprisingly tolerant of this. Most of a weight matrix’s values cluster in a narrow range, and neural networks in general have enough redundancy that small per-weight errors average out rather than compounding into obviously broken output. A well-executed 4-bit quantisation loses noticeably less quality than you’d naively expect from throwing away three-quarters of the bits.

That tolerance has limits, though, and where those limits sit depends heavily on how the quantisation is done, which is where the format differences actually matter.

GGUF: quantisation for CPU-friendly, portable inference

GGUF is llama.cpp’s model format, and it’s the one most homelab users meet first because it’s what Ollama uses under the hood. It bundles the quantised weights, tokenizer, and metadata into a single portable file, and it supports a wide range of quantisation schemes denoted by names like Q4_K_M, Q5_K_S, or Q8_0.

The naming isn’t arbitrary. The number is the bit width — Q4 is roughly 4 bits per weight, Q8 roughly 8. The letter suffix after the underscore (K, or the older 0/1) indicates the quantisation method’s sophistication — K-quants use variable bit allocation per block, spending more bits on weights that matter more to output quality and fewer on ones that don’t, which is why Q4_K_M reliably outperforms an older flat Q4_0 at the same nominal bit width. The final letter — S, M, L for small/medium/large — is a rough size/quality dial within that scheme; M is the sensible default for most people, S if you’re VRAM-desperate, L if you have room to spare and want to claw back a little quality.

GGUF’s real strength is that it runs everywhere — CPU-only, partial GPU offload, full GPU offload — from the same file, which makes it the natural choice for anyone whose hardware situation isn’t a clean “big dedicated GPU.”

GPTQ and AWQ: quantisation for GPU-only serving

GPTQ and AWQ are a different family, designed specifically for GPU inference and used by servers like vLLM rather than llama.cpp. Both are post-training quantisation methods that use a calibration dataset to decide how to minimise error, rather than just rounding weights blindly.

GPTQ quantises layer by layer, adjusting remaining weights to compensate for the error introduced by quantising earlier ones — a more careful, mathematically involved process than GGUF’s simpler block-wise approach, and one that historically produced strong results at 4-bit specifically. AWQ (Activation-aware Weight Quantisation) takes a different angle: it identifies which weights matter most based on the activations they interact with during calibration, and protects that smaller set of salient weights from precision loss while quantising the rest more aggressively. In practice AWQ tends to edge out GPTQ at the same bit width, and it’s become the more commonly recommended choice for new GPU-only deployments.

The trade-off against GGUF is portability: GPTQ and AWQ models are built for full GPU residency and don’t give you llama.cpp’s graceful CPU-offload fallback. If you’re serving through vLLM on a dedicated GPU box, that’s not a real cost. If your hardware situation is more mixed, GGUF’s flexibility usually wins.

Choosing a bit width

The practical question is rarely “which format” in isolation — it’s “what fits, and what do I lose.” A rough mental model that’s held up across the models I’ve actually run:

Q8 / 8-bit is close to indistinguishable from full precision for most tasks, at roughly half the memory of FP16. Worth using if you have the VRAM headroom and want the smallest possible quality risk.

Q4_K_M / 4-bit is the practical sweet spot for most homelab setups — a quarter of the FP16 footprint with quality degradation that’s genuinely hard to notice on general chat, summarisation, and coding-assistance tasks. This is where I run almost everything.

Q3 and below starts to show real cracks — noticeably worse instruction-following, more hallucination on factual recall, occasional grammatical breakdown on longer generations. I’d only reach for this if a model literally doesn’t fit any other way, and I’d sanity-check the output against the same prompts run unquantised before trusting it for anything that matters.

Fitting a model in VRAM is arithmetic once you know the bit width: parameters times bits, divided by 8, plus roughly 10–20% overhead for the context window’s KV cache and inference scratch space.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

7B model, Q4_K_M (~4.5 bits/weight effective):
  7,000,000,000 × 4.5 / 8 ≈ 3.9GB weights
  + ~1–2GB context/overhead
  ≈ 5–6GB total — comfortably fits an 8GB card

13B model, Q4_K_M:
  13,000,000,000 × 4.5 / 8 ≈ 7.3GB weights
  + ~2GB overhead
  ≈ 9–10GB — fits a 12GB card, tight on 8GB

70B model, Q4_K_M:
  70,000,000,000 × 4.5 / 8 ≈ 39GB weights
  + overhead
  ≈ 42–45GB — needs a 48GB card, or CPU/GPU split via llama.cpp

What calibration data actually changes

It’s worth being specific about why GPTQ and AWQ need a calibration dataset at all, because it’s a genuine difference from GGUF’s simpler approach and it explains a failure mode you’ll otherwise find confusing. Both methods run a small batch of representative text through the unquantised model first, recording how activations actually behave layer by layer. GPTQ uses that pass to solve for weight adjustments that minimise the output error introduced at each layer; AWQ uses it to identify which weight channels see the largest, most important activations and protects those specifically.

The practical consequence is that calibration data choice matters more than people expect. A model calibrated on general web text and then used almost exclusively for, say, code generation can end up quantised in a way that’s slightly mismatched to its actual workload — the calibration pass simply never saw much code, so it didn’t know to protect the weight channels that matter most for that domain. This is why some community-published quantisations explicitly note their calibration set (wikitext, a code-heavy mix, a multilingual set) — if your use case is narrow and specific, it’s worth checking that the quant you’re downloading was calibrated on something reasonably close to it, rather than assuming all GPTQ or AWQ builds of the same base model are functionally identical.

GGUF’s K-quants, by contrast, don’t use a calibration dataset in the same sense — the variable bit allocation is based on statistical properties of the weights themselves (which tend to be domain-agnostic), which is part of why GGUF quants are more consistent across different downstream uses even though they’re a comparatively simpler technique.

Mixing precision within one model

A detail that surprises people coming to this fresh: quantisation is rarely applied uniformly across an entire model even within a single named quant level. Certain layers — the embedding layer, the final output projection, sometimes the attention layers as distinct from the feed-forward layers — are disproportionately sensitive to precision loss relative to their size, and most serious quantisation implementations know this and treat them specially. GGUF’s K_M and K_L variants specifically keep a subset of tensors at a higher bit width than the nominal quant level suggests, which is exactly why Q4_K_M outperforms a hypothetical flat 4-bit-everywhere quantisation at the same file size — it isn’t actually 4 bits everywhere, it’s 4 bits where that’s cheap and a bit more where it matters. Understanding this explains why two quants with similar file sizes but different internal precision distribution can produce noticeably different output quality even though the headline number is the same.

Running a GGUF model with llama.cpp

Picking a quantisation level in practice is usually just choosing the right file from a model repository:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# download a specific quant level directly
huggingface-cli download bartowski/Meta-Llama-3.1-8B-Instruct-GGUF \
  Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  --local-dir ./models

# run it
./llama-server \
  -m ./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  --n-gpu-layers 999 \
  --ctx-size 8192 \
  --port 8080

--n-gpu-layers 999 tells llama.cpp to offload as many transformer layers to GPU as will fit, falling back to CPU for the rest — the graceful degradation that’s GGUF’s whole point. Drop it lower deliberately if you want to reserve VRAM for something else running on the same card.

Troubleshooting

Model quality feels noticeably worse than the same model’s reputation online suggests. Check the actual quant level you downloaded — repositories often default their “recommended” download to a smaller quant than you’d choose given your VRAM, to be broadly compatible. Re-check against your actual headroom; you may have room for Q5_K_M or Q6_K instead of Q4_K_M.

GGUF model loads but generation is much slower than expected despite a capable GPU. --n-gpu-layers is probably too low, leaving most layers on CPU. Check the server’s startup log — llama.cpp prints how many layers actually landed on GPU versus CPU, and it’s easy to have left a conservative default in place from an earlier, more VRAM-constrained model.

GPTQ/AWQ model fails to load in vLLM with a dtype or quantisation-method mismatch error. vLLM needs to be told explicitly which quantisation method the checkpoint uses — pass --quantization awq or --quantization gptq matching the model card; it won’t reliably auto-detect from the file alone across all model repos.

Same nominal bit width, wildly different file sizes between two GGUF downloads. Group size and the specific K-quant variant both affect the real bits-per-weight, which only approximates the number in the name. Compare file sizes directly rather than trusting the label alone if VRAM budget is tight to the gigabyte.

Quantised model gives confidently wrong factual answers it didn’t give before. This is quantisation eating into the model’s least-robust behaviour first — factual recall tends to degrade before fluency does. If this matters for your use case, step up a bit width rather than accepting it as a fixed cost of running locally.

Verdict

Quantisation is not a compromise to feel bad about — it’s the specific piece of engineering that makes self-hosted LLMs practical on consumer hardware at all, and a well-chosen 4-bit GGUF or AWQ model is close enough to full precision for the overwhelming majority of homelab use cases that the memory savings are close to free. The place to be deliberate is matching the format to your serving stack — GGUF for llama.cpp’s CPU-friendly flexibility, GPTQ or AWQ for GPU-only servers like vLLM — and picking a bit width based on your actual VRAM rather than grabbing whatever a repository lists first.