Quantisation Explained: Why Q4_K_M Is Usually Enough

You go to download a model and the page offers you nineteen files. Q2_K, Q3_K_S, Q3_K_M, Q3_K_L, Q4_0, Q4_1, Q4_K_S, Q4_K_M, Q5_0, Q5_K_M, Q6_K, Q8_0, and an F16 that’s four times the size of any of them. There’s a table of file sizes and a column labelled “perplexity” with numbers that differ in the second decimal place. Nothing tells you which one to click.

The short answer is Q4_K_M. It’s been the right default for months and I’d guess it stays the right default for a while yet. The longer answer is why, and it’s worth ten minutes because once the naming scheme stops being noise you can make this choice deliberately rather than by folklore.

What quantisation is doing

A model’s weights are numbers. Trained in 16-bit floating point, each weight costs 2 bytes, so a 7-billion-parameter model is roughly 13.5 GB of pure weights before you’ve allocated a single byte for context. That’s the file you can’t fit on your graphics card.

Quantisation stores those numbers with fewer bits. Take the naive approach first, because the naive approach is nearly right: pick the largest weight in the model, divide everything by it, multiply by 15, round to the nearest integer. Now every weight is a 4-bit number between 0 and 15 and the file is a quarter of the size.

This produces garbage, for a reason that’s obvious once stated. Weights in a neural network are mostly small and clustered near zero, with a few large outliers. Scale everything by the largest one and almost every weight rounds to 0, 1 or 2 — you’ve thrown away the distinctions that mattered and preserved resolution in a range where nothing lives.

The fix is block-wise scaling. Chop the weights into small blocks — 32 of them — and give each block its own scale factor derived from its own largest value. A block of tiny weights gets a tiny scale and keeps its precision; the block containing an outlier gets a big scale and eats the error locally. That’s Q4_0: 32 weights at 4 bits each (16 bytes), plus one 16-bit scale (2 bytes), so 18 bytes per 32 weights, which works out at 4.5 bits per weight rather than the advertised 4.

Q4_1 adds a per-block minimum as well as a scale, so blocks whose values cluster away from zero get a tighter fit. It costs another 2 bytes per block, which is where the 5 bits per weight comes from.

What the K means, and why it changed everything

The formats above treat every weight in the model identically. That’s a strange assumption if you’ve looked at what a transformer layer contains, because the parts have wildly different sensitivity to rounding error. Squash the attention output projection and the model degrades sharply; squash a feed-forward layer’s up-projection by the same amount and it barely notices.

K-quants, which landed in llama.cpp in mid-2023, exploit this. Two mechanisms:

Super-blocks. Instead of one flat scale per block of 32, k-quants group 8 or 16 blocks into a super-block and quantise the scales themselves. The per-block scales are stored at reduced precision relative to a super-block scale. You spend fewer bits on metadata and can afford more granular blocks with the savings.

Mixed precision by tensor role. This is the part that matters, and it’s why the S/M/L suffix exists. A Q4_K_M model does not store everything at 4 bits. It stores most tensors at Q4_K and deliberately promotes the ones that hurt most when damaged — the attention value projections and the feed-forward down-projections — to Q6_K. The _S, _M, _L suffixes are Small, Medium and Large, describing how generous that promotion is.

So Q4_K_M decodes as: 4-bit weights, K-quant scheme, Medium mix — meaning “mostly 4-bit, with the sensitive tensors kept at 6-bit”. It’s a hand-tuned compromise that somebody measured carefully, and the measurement is the reason it wins.

The numbers

Perplexity measures how surprised a model is by text it should be able to predict. Lower is better, and the useful reading is the gap from the unquantised original rather than the absolute value. Figures below are the well-travelled llama.cpp measurements on a 7B model, which have held their shape across model families:

Read three things off that table.

Q8_0 is pointless. It doubles Q4_K_M’s size to recover five hundredths of a point of perplexity. Nobody can perceive that in output. It exists for people quantising from Q8 or doing measurements, and it clutters download pages.

Q4_K_M beats Q4_0 decisively at nearly the same size — 3.80 GB versus 3.50 GB, and the perplexity gap between them is five times larger than the gap between Q4_K_M and F16. Every old Q4_0 GGUF on a download page is a legacy artefact. Skip it.

Q2_K falls off a cliff. That +0.87 is visible in output: coherence survives, instruction-following gets sloppy, the model starts losing track of what you asked. Q3_K_M sits at the edge of acceptable and I reach for it only when a model is otherwise unrunnable.

The argument that actually decides it

Here’s the thing that reframes the whole choice. Stop asking “which quantisation of this model?” and ask “what’s the best model that fits in N gigabytes?”

At around 7 GB of disk, your options include a 7B at Q8_0 (6.7 GB, perplexity 5.9070) and a 13B at Q4_K_M (7.87 GB, perplexity 5.0898 on the same test). The 13B is dramatically better — a whole point of perplexity, which is the difference between model generations, and it costs you 1.2 GB.

The heavily-quantised bigger model wins, and it keeps winning until you get down around Q3, where the damage starts outrunning the benefit of the extra parameters. This is the single most useful fact in this article: spend your VRAM on parameters and pay for them with bits. People agonise over Q5 versus Q6 on a 7B while a 13B at Q4_K_M sits there being better than both.

The exception is small models. A 3B at Q4_K_M degrades noticeably more than a 7B at the same format, because there’s less redundancy in the weights to absorb the rounding error. Below about 7B, step up to Q5_K_M or Q6_K — the file is small enough that you can afford it.

Rolling your own

You mostly don’t need to; the community quantises everything within hours of release. When you’ve fine-tuned something yourself, the tooling is two commands:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

cd llama.cpp

# HF safetensors -> GGUF at fp16
python convert.py /models/my-finetuned-7b/ \
  --outfile /models/my-finetuned-7b/f16.gguf \
  --outtype f16

# fp16 GGUF -> Q4_K_M
./quantize /models/my-finetuned-7b/f16.gguf \
  /models/my-finetuned-7b/my-finetuned-7b.Q4_K_M.gguf \
  Q4_K_M

Then confirm it survived, with a perplexity run against a text file you keep for the purpose:

1
2
3

./perplexity -m /models/my-finetuned-7b/my-finetuned-7b.Q4_K_M.gguf \
  -f /models/wiki.test.raw \
  --ctx-size 512

Compare that number against the same run on the f16. A gap in the region of 0.05 means the quantisation went as expected. A gap of 0.5 means something is wrong with the conversion, and the usual culprit is an unusual architecture that convert.py handled approximately.

The fp16 intermediate is the annoying part: quantising a 13B means having ~26 GB of disk free and reading all of it. Delete it afterwards.

Where 4 bits genuinely isn’t enough

Perplexity is one number over a corpus of English prose, and it hides task-specific damage. Four places where I’ve watched Q4_K_M underperform its perplexity score:

Code generation. Code is unforgiving in a way prose isn’t. A summary with a slightly wrong word is still a good summary; a function with a slightly wrong variable name doesn’t run. Quantisation errors show up as subtly incorrect identifiers, off-by-one indices, and imports that don’t exist. I keep code models a rung higher — Q5_K_M at minimum — and the difference is visible within an afternoon of use.

Languages other than English. Perplexity gets measured on English text, so the tokens that carry the measurement are the ones the model saw most during training and represents most robustly. Lower-resource languages sit in the thinner parts of the weight distribution, and that’s exactly where rounding error does its damage. Danish output from a Q4 model reads noticeably more stilted than its English, at a perplexity delta the table would call negligible.

Long structured output. Asking for 300 lines of valid YAML compounds every small error, since a single misplaced indent invalidates everything after it. The higher quant buys fewer retries. If the output must parse, constrain it at the sampler with a grammar rather than paying for bits — llama.cpp’s GBNF support makes the guarantee structural.

Heavily fine-tuned models. A LoRA merged into a base model concentrates its changes in a small number of weights with unusual magnitudes. Those are the outliers that block-wise scaling handles worst. Quantise your own fine-tune to Q4 and measure it against the fp16 on your task, since the community’s numbers describe base models and yours is no longer one.

None of these makes Q4_K_M the wrong default. They make it a default worth overriding deliberately, which is a different thing, and the override is cheap: download the Q5, run the same prompts, keep whichever one you can’t distinguish from the other.

There’s also a speed dimension people forget. The k-quant’s mixed precision means the dequantisation step touches two different formats per layer, so on a pure-CPU run you’re doing measurably more work per token than a flat Q4_0 would. On a GPU it vanishes into the memory bandwidth — the card spends its life waiting for weights to arrive from VRAM, and a smaller file arriving faster more than pays for the extra arithmetic. This is why inference on a CPU behaves so differently from the same model on a card: the bottleneck moves, and the format that wins moves with it.

Troubleshooting

“unknown model architecture” or a flat refusal to load. GGUF replaced GGML in August 2023 and old .bin GGML files don’t load in current builds. Re-download the GGUF, or check the file actually starts with the four bytes GGUF:

1
2

$ head -c 4 model.gguf | xxd
00000000: 4747 5546                                GGUF

Output is fluent nonsense — grammatical, confident, wrong. Usually the wrong prompt template rather than the quantisation. An instruct model given raw text instead of its expected [INST] wrapper behaves exactly like this. Test the same prompt against a higher-bit quant of the same model before you blame the bits.

Output degrades as the conversation gets longer. That’s the KV cache, which is separate from the weights and stored at fp16 by default. Long context at 4-bit weights can mean the cache costs more VRAM than the model. Shorten the context before you re-quantise anything.

Loading is slow, and the machine swaps. llama.cpp memory-maps the GGUF, so pages arrive from disk on demand. On a spinning disk the first inference crawls while the file streams in, then it’s fine. On a machine with less RAM than the model, it never becomes fine.

A Q4_K_M runs slower than a Q4_0 on the same hardware. Real, and expected on CPU: the k-quant’s mixed precision means more work per block during dequantisation. You’re paying maybe 5–10% throughput for a large quality gain, which is a trade I take every time.

The verdict

Download the Q4_K_M. If the model is under 7B, take Q5_K_M instead. If Q4_K_M leaves several gigabytes of VRAM unused, don’t spend them on Q6 of the same model — go and get the next size up and run that at Q4_K_M.

The formats below Q4 are for when the alternative is running nothing at all, and that’s a legitimate situation: a 34B at Q3_K_M on a 24 GB card is a real capability that doesn’t otherwise exist for you. Just know you’re trading rather than optimising.

Everything above Q5 is for measurement, archival, or as the source for further quantisation. I’ve never kept a Q8 on a running machine and never missed it — which is a good illustration of why the engine’s internals are worth understanding: once you know that a block of 32 weights shares a scale factor, the whole naming scheme stops being alphabet soup and starts being a menu with prices on it. If you want the wider view across formats — GPTQ, AWQ and the GPU-side ecosystem — that comparison lives here.