What is quantization?
By TryRAG · Updated
Try it: the Quantization Visualizer rounds a sample of weights to 8, 4 or 3 bits and shows the error on each one. Its second tab picks a GGUF quantization for your memory.
Quantization means storing each number in a model with fewer bits. A model trained with 16-bit weights and stored in 4 bits needs about a quarter of the memory, so it fits on a smaller GPU or an ordinary laptop. The price is a small error on every weight.
How it works
Four bits can hold 16 different values. To store a group of weights in 4 bits:
- Find the largest weight in the group.
- Choose a scale, the size of one step, so that the largest weight just fits within the available steps.
- Replace each weight with the whole number of steps nearest to it.
When the model runs, each stored number is multiplied by the scale to get the weight back. It comes back as the nearest step, not the original value. The difference is the rounding error, and it is never more than half a step.
Why fewer bits hurt more than you might expect
Every bit removed halves the number of steps, which doubles the step size and so doubles the error. Going from 8 bits to 4 makes the typical error about sixteen times larger. Going from 4 to 3 doubles it again. That is why 8-bit models are almost indistinguishable from the original, 4-bit models lose a little, and below 4 bits quality drops quickly.
The outlier problem
The scale is set by the largest weight. Most weights in a layer are close to zero, but a few are much larger. If one weight is ten times bigger than the rest, the steps must be ten times coarser to reach it, and all the ordinary weights end up sharing two or three steps. One weight spoils the precision of thousands.
Block-wise scaling
The answer used by practical formats is to give small blocks of weights, often 32 at a time, their own scale. A large weight then affects only its own block. The extra scales have to be stored, which is why real "4-bit" files work out at 4.5 to 5 bits per weight.
Choosing a GGUF quantization
GGUF is the file format used by llama.cpp and the tools built on it. Its names give the bits and the variant. These sizes are for an 8-billion-parameter model and are approximate:
| Name | Bits per weight | File size | Quality |
|---|---|---|---|
| Q8_0 | about 8.5 | about 7.9 GiB | Nearly identical to the original |
| Q5_K_M | about 5.7 | about 5.3 GiB | Very small loss |
| Q4_K_M | about 4.8 | about 4.5 GiB | The usual default |
| Q3_K_M | about 3.9 | about 3.7 GiB | Noticeable loss |
- Pick the largest file that fits your memory with room left for the context. The KV Cache Calculator shows how much the context needs.
- If only a 3-bit file of a large model fits, try a smaller model at 4 or 5 bits. It is often better.
- Test on your own questions. The loss from quantization varies between models and tasks.
Words you will meet
- FP16, INT8, INT4: 16-bit floating point, and 8-bit and 4-bit whole numbers.
- Scale: the size of one step for a group of weights.
- Symmetric: steps centred on zero. Asymmetric: steps fitted between the smallest and the largest weight.
- Bits per weight: the file size in bits divided by the number of weights, including the stored scales.