Quantisation Explained: GGUF, GPTQ and Small VRAM

Tue, 14 Oct 2025 08:45:00 +0000

The first time I tried to run a 13B model on a card with 12GB of VRAM, I hit an out-of-memory error before the model had finished loading. The maths is unforgiving: a 13-billion-parameter model in the FP16 precision it was likely trained and released in needs roughly 26GB just to hold the weights, before you’ve allocated a single byte for the context window. My card had less than half that. Quantisation is the technique that made it possible anyway, and understanding what it actually does — rather than treating it as a magic “make it smaller” checkbox — is the difference between picking a sensible trade-off and quietly running a model that’s dumber than it needs to be.

Quantisation - vo.rs

Quantisation Explained: GGUF, GPTQ and Small VRAM