KV Cache Calculator
See how much GPU memory a model needs for its weights and its KV cache, and why long context and big batches fill a GPU. Nothing is uploaded.
- 1Pick a model and a GPU
- 2Move the context and batch sliders
- 3See whether it fits and what fills the memory
Model and GPU
Workload
Assumptions (change if you know better)
The cache against the weights as context grows
What GQA saves
| Attention | KV heads | Cache per token | Cache for this workload |
|---|
How it is worked out
What it is for
- Check whether a model, a context length and a batch size fit on a GPU before renting it.
- See how the KV cache grows with context length and with the number of conversations served at once.
- Understand why grouped-query attention (GQA) and lower cache precision exist.
- See how much memory retrieval adds when RAG chunks are put into every prompt.
How it works
When a language model writes a token it looks back at everything before it. To avoid redoing that work for every new token, it keeps two vectors per token for every layer: a key and a value. This store is the KV cache. It saves a great deal of computation and costs memory.
The size is simple to work out. For each token the cache holds 2 × layers × KV heads × head dimension values. Multiply by the bytes per value, the context length and the number of sequences in the batch. For Llama 3 8B at 16-bit precision that is 128 KiB per token, or about 1 GiB for one conversation of 8,192 tokens.
The weights are loaded once and shared by every request. The cache is not shared: every conversation in a batch has its own. That is why a model that fits easily for one user can run out of memory at a batch of eight, and why the cache can end up larger than the model.
The calculator adds the weights, the cache and a fixed allowance for everything else on the GPU, and compares the total with the GPU memory. It is an estimate for planning. Real servers also reserve memory for activations and fragmentation, and some reserve the whole cache up front.
Questions
What is the KV cache?
It is the model's stored working for the text so far: a key and a value vector for every token in every layer. With it, producing the next token needs one pass over the new token only, not over the whole conversation again.
Why does the cache grow with batch size but the weights do not?
The weights are the model itself and are the same for every request. The cache holds what the model has worked out about one particular conversation, so each conversation needs its own.
What is GQA?
Grouped-query attention lets several attention heads share one set of keys and values. Llama 3 8B has 32 heads but only 8 sets, so its cache is a quarter of the size it would otherwise be. Tick "Turn off GQA" to see the difference.
Are sizes in GB or GiB?
In GiB, where 1 GiB is 1,073,741,824 bytes. GPU memory is sold in these units: a 24 GB card has 24 GiB. Model sizes on download pages are often in GB of 1,000,000,000 bytes, which makes them look about 7% larger.
What is "other GPU memory"?
An allowance for what the calculator does not model: the inference program itself, temporary values while a token is computed, and gaps left by fragmentation. 1.5 GiB is a rough starting point. Change it under Assumptions.
Can I enter a model that is not listed?
Yes. Choose Custom and enter the number of layers, attention heads, KV heads, the head dimension and the parameter count. They are in the model's config.json file.
Can I share my settings?
Yes. The settings are kept in the page address, so Copy link gives a link that opens the page exactly as you left it. CSV and JSON download the inputs and the results.
Is anything uploaded?
No. The page is a calculator that runs in your browser tab. The numbers you enter stay on your device.