Quantization Visualizer

See what storing weights in 8, 4 or 3 bits does to them, why one large weight can spoil the rest, and which GGUF quantization fits your memory. Nothing is uploaded.

Quantization

Weights before and after

Rounding error on each weight

The same bits at each group size

Group sizeMean squared errorLargest errorBits per weightOf FP16 size

How it is worked out

    What it is for

    How it works

    A model's weights are usually trained as 16-bit or 32-bit numbers. Quantization stores each one in fewer bits, most often 8 or 4, so the model takes less memory and can run on smaller hardware.

    With 4 bits there are only 16 different numbers to store. The weights in a group are measured, a scale is chosen so the largest weight just fits, and every weight is rounded to the nearest step of that scale. The difference between the original and the rounded value is the rounding error.

    The weakness is the largest weight. If one weight is ten times bigger than the others, the scale has to be ten times coarser to reach it, and all the ordinary weights are squeezed onto two or three steps. Tick "Add one large weight" to see it.

    The fix used by real formats is to scale small blocks separately. With a scale for every 32 weights, a large weight only harms its own block. The cost is the storage for the extra scales, which is why a "4-bit" file is really about 4.5 to 5 bits per weight.

    The sample here is 512 random weights with a bell-shaped spread, like one small piece of a real layer. The picture is real quantization arithmetic, but it shows the error on weights, not the effect on a model's answers, which depends on the model and the method.

    Questions

    What is quantization?

    Storing each weight of a model in fewer bits than it was trained with. A 16-bit model stored in 4 bits takes about a quarter of the memory, at the price of small rounding errors on every weight.

    What is the difference between symmetric and asymmetric?

    Symmetric quantization centres the levels on zero and needs only a scale. Asymmetric fits the levels between the smallest and the largest weight, which uses the range better when the weights are lopsided, and needs an offset as well as a scale.

    What is group size?

    How many weights share one scale. "Whole tensor" uses a single scale for all of them. Blocks of 128, 64 or 32 each get their own, which follows the weights more closely and costs a little more storage.

    Why is a 4-bit GGUF file more than 4 bits per weight?

    Each block also stores its scale, and the K-quant formats keep the most sensitive layers at higher precision. Q4_K_M works out at roughly 4.8 bits per weight.

    Which GGUF quantization should I use?

    The largest one that fits your memory with room left for the context. Q4_K_M is the usual starting point. Below Q4 the loss of quality becomes noticeable, and a smaller model at higher precision is often better.

    How accurate is the GGUF picker?

    It multiplies the parameter count by an approximate number of bits per weight for each format. Real files differ by a few percent between models. Treat it as an estimate and check the file size on the download page.

    Can I share my settings?

    Yes. The settings are kept in the page address, so Copy link gives a link that opens the page exactly as you left it. CSV and JSON download the inputs and the results.

    Is anything uploaded?

    No. The page is a calculator that runs in your browser tab. The numbers you enter stay on your device.