What is quantization?

By TryRAG · Updated

Try it: the Quantization Visualizer rounds a sample of weights to 8, 4 or 3 bits and shows the error on each one. Its second tab picks a GGUF quantization for your memory.

Quantization means storing each number in a model with fewer bits. A model trained with 16-bit weights and stored in 4 bits needs about a quarter of the memory, so it fits on a smaller GPU or an ordinary laptop. The price is a small error on every weight.

How it works

Four bits can hold 16 different values. To store a group of weights in 4 bits:

  1. Find the largest weight in the group.
  2. Choose a scale, the size of one step, so that the largest weight just fits within the available steps.
  3. Replace each weight with the whole number of steps nearest to it.

When the model runs, each stored number is multiplied by the scale to get the weight back. It comes back as the nearest step, not the original value. The difference is the rounding error, and it is never more than half a step.

Why fewer bits hurt more than you might expect

Every bit removed halves the number of steps, which doubles the step size and so doubles the error. Going from 8 bits to 4 makes the typical error about sixteen times larger. Going from 4 to 3 doubles it again. That is why 8-bit models are almost indistinguishable from the original, 4-bit models lose a little, and below 4 bits quality drops quickly.

The outlier problem

The scale is set by the largest weight. Most weights in a layer are close to zero, but a few are much larger. If one weight is ten times bigger than the rest, the steps must be ten times coarser to reach it, and all the ordinary weights end up sharing two or three steps. One weight spoils the precision of thousands.

Block-wise scaling

The answer used by practical formats is to give small blocks of weights, often 32 at a time, their own scale. A large weight then affects only its own block. The extra scales have to be stored, which is why real "4-bit" files work out at 4.5 to 5 bits per weight.

Choosing a GGUF quantization

GGUF is the file format used by llama.cpp and the tools built on it. Its names give the bits and the variant. These sizes are for an 8-billion-parameter model and are approximate:

NameBits per weightFile sizeQuality
Q8_0about 8.5about 7.9 GiBNearly identical to the original
Q5_K_Mabout 5.7about 5.3 GiBVery small loss
Q4_K_Mabout 4.8about 4.5 GiBThe usual default
Q3_K_Mabout 3.9about 3.7 GiBNoticeable loss

Words you will meet