Quantization Visualizer
See what storing weights in 8, 4 or 3 bits does to them, why one large weight can spoil the rest, and which GGUF quantization fits your memory. Nothing is uploaded.
- 1Choose 8, 4 or 3 bits
- 2Add one large weight
- 3Shrink the group size and watch the error fall
Quantization
Weights before and after
Rounding error on each weight
The same bits at each group size
| Group size | Mean squared error | Largest error | Bits per weight | Of FP16 size |
|---|
How it is worked out
Your model and memory
Estimates. File sizes use approximate bits per weight for each format and vary a little between models.
| Quantization | Bits per weight (approx.) | File size | Memory needed | Fits | Quality |
|---|
For the memory the context itself needs, see the KV Cache Calculator.
What it is for
- See what quantization does to individual weights, not only to the file size.
- Understand why an outlier hurts and why real formats scale small blocks of weights separately.
- Compare error and size across group sizes at the same number of bits.
- Choose a GGUF quantization that fits the memory you have.
How it works
A model's weights are usually trained as 16-bit or 32-bit numbers. Quantization stores each one in fewer bits, most often 8 or 4, so the model takes less memory and can run on smaller hardware.
With 4 bits there are only 16 different numbers to store. The weights in a group are measured, a scale is chosen so the largest weight just fits, and every weight is rounded to the nearest step of that scale. The difference between the original and the rounded value is the rounding error.
The weakness is the largest weight. If one weight is ten times bigger than the others, the scale has to be ten times coarser to reach it, and all the ordinary weights are squeezed onto two or three steps. Tick "Add one large weight" to see it.
The fix used by real formats is to scale small blocks separately. With a scale for every 32 weights, a large weight only harms its own block. The cost is the storage for the extra scales, which is why a "4-bit" file is really about 4.5 to 5 bits per weight.
The sample here is 512 random weights with a bell-shaped spread, like one small piece of a real layer. The picture is real quantization arithmetic, but it shows the error on weights, not the effect on a model's answers, which depends on the model and the method.
Questions
What is quantization?
Storing each weight of a model in fewer bits than it was trained with. A 16-bit model stored in 4 bits takes about a quarter of the memory, at the price of small rounding errors on every weight.
What is the difference between symmetric and asymmetric?
Symmetric quantization centres the levels on zero and needs only a scale. Asymmetric fits the levels between the smallest and the largest weight, which uses the range better when the weights are lopsided, and needs an offset as well as a scale.
What is group size?
How many weights share one scale. "Whole tensor" uses a single scale for all of them. Blocks of 128, 64 or 32 each get their own, which follows the weights more closely and costs a little more storage.
Why is a 4-bit GGUF file more than 4 bits per weight?
Each block also stores its scale, and the K-quant formats keep the most sensitive layers at higher precision. Q4_K_M works out at roughly 4.8 bits per weight.
Which GGUF quantization should I use?
The largest one that fits your memory with room left for the context. Q4_K_M is the usual starting point. Below Q4 the loss of quality becomes noticeable, and a smaller model at higher precision is often better.
How accurate is the GGUF picker?
It multiplies the parameter count by an approximate number of bits per weight for each format. Real files differ by a few percent between models. Treat it as an estimate and check the file size on the download page.
Can I share my settings?
Yes. The settings are kept in the page address, so Copy link gives a link that opens the page exactly as you left it. CSV and JSON download the inputs and the results.
Is anything uploaded?
No. The page is a calculator that runs in your browser tab. The numbers you enter stay on your device.