What is Quantisation?

The plain definition

Model weights are normally trained in 32-bit or 16-bit floating point. Quantisation stores them in fewer bits — 8-bit integers, or 4-bit for aggressive compression. A model held at 4 bits instead of 16 occupies roughly a quarter of the memory.

That reduction matters for two reasons. It determines whether a model fits on the hardware you have at all, which is often the difference between one expensive GPU and none. And because inference on large models is typically limited by memory bandwidth rather than arithmetic, moving fewer bytes makes generation faster as well as cheaper.

The methods, and what they cost

Post-training quantisation converts an already-trained model. It needs no retraining and, at 8 bits, typically costs very little accuracy. This is the default and usually sufficient.

Calibrated methods such as GPTQ and AWQ use a small representative dataset to decide how to allocate precision, preserving the weights that matter most. They make 4-bit viable with modest degradation, which pure naive rounding does not.

Quantisation-aware training simulates low precision during training so the model learns weights robust to it. It gives the best quality at very low bit widths and requires a training run, so it is reserved for cases where the deployment constraint is severe.

QLoRA combines the ideas from the other direction: hold a frozen base model in 4-bit while training LoRA adapters in higher precision, which is what makes fine-tuning sizeable models on a single GPU practical.

How to decide, and where it bites

The rule of thumb: 8-bit is close to free and should be the starting assumption. 4-bit is usually a good trade and needs verification. Below 4 bits, degradation becomes difficult to ignore for general-purpose use.

The important discipline is measuring the loss on your task rather than trusting a published benchmark. Aggregate benchmark scores hide uneven damage: quantisation tends to degrade long-chain reasoning, arithmetic, precise instruction-following, and rarely-seen knowledge more than it degrades fluent conversational output. A model can look nearly unchanged on a summarisation benchmark while getting materially worse at the structured extraction your product depends on. Run your own evaluation set at each precision and compare.

The other genuine trade-off is capacity versus precision at fixed memory. Given a fixed GPU, a larger model at 4 bits usually beats a smaller model at 16 bits — parameter count tends to matter more than per-weight fidelity. That is worth testing rather than assuming, but it is the more common outcome, and it reframes quantisation as a way to run better models on the hardware you have rather than merely a cost saving.