What is the KV cache?
By TryRAG · Updated
Try it: the KV Cache Calculator works out the cache for a model, a context length and a batch size, and shows whether it fits on a GPU.
A language model writes one token at a time. To choose each token it looks back at every token before it. The KV cache is the working it keeps from those earlier tokens so that it does not have to redo it. It makes generation fast, and it is the reason long conversations and busy servers run out of GPU memory.
What is stored
Inside each layer of the model, every token is turned into two vectors: a key, which says what the token can be matched on, and a value, which is what it contributes when matched. A new token is compared with the keys of all earlier tokens and collects their values. The keys and values of earlier tokens never change, so they are computed once and kept. That store, for every layer and every token so far, is the KV cache.
Without it, writing token 1,000 would mean processing all 999 earlier tokens again. With it, only the new token is processed.
How big it is
For one token the cache holds:
2 × layers × KV heads × head dimension × bytes per value
The 2 is for keys and values. For Llama 3 8B that is 2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes, or 128 KiB for every token. Multiply by the context length and by the number of conversations being served together:
| Llama 3 8B, 16-bit | Cache |
|---|---|
| 1 conversation, 2,048 tokens | 0.25 GiB |
| 1 conversation, 8,192 tokens | 1 GiB |
| 8 conversations, 8,192 tokens each | 8 GiB |
| 32 conversations, 8,192 tokens each | 32 GiB |
The weights of the same model take about 15 GiB and do not grow. At a batch of 16 the cache is already larger than the model.
Why batches multiply it
The weights are the model, and one copy serves every request. The cache belongs to one conversation: it holds what the model has worked out about that particular text. Eight conversations need eight caches. This is why a model that fits comfortably for a single user can fail to fit when a server tries to handle several at once.
Three ways it is made smaller
- Grouped-query attention (GQA). Several attention heads share one set of keys and values. Llama 3 8B has 32 heads and 8 sets, so its cache is a quarter of what it would be. Most recent models use GQA.
- Lower precision. Storing each cached value in 8 bits instead of 16 halves the cache, with a small effect on quality.
- Paged allocation. Giving each conversation memory in small blocks as it grows, instead of reserving the maximum in advance, lets more conversations share the same GPU. The Batching Simulator shows the difference.
What it means for RAG
Retrieved chunks are part of the prompt, so they are part of the cache. Five chunks of 300 words add about 2,000 tokens to every request, which for Llama 3 8B is about 250 MiB of cache per conversation. Retrieving fewer or shorter chunks saves GPU memory as well as API cost. The chunk size guide covers how to choose.
Words you will meet
- Context length: the tokens held for one conversation, prompt and answer together.
- Head dimension: the length of each key and value vector.
- Prefill: reading the prompt, which fills the cache before the first token is written.
- GiB: 1,073,741,824 bytes, the unit GPU memory is sold in.