What is the KV cache?

By TryRAG · Updated

Try it: the KV Cache Calculator works out the cache for a model, a context length and a batch size, and shows whether it fits on a GPU.

A language model writes one token at a time. To choose each token it looks back at every token before it. The KV cache is the working it keeps from those earlier tokens so that it does not have to redo it. It makes generation fast, and it is the reason long conversations and busy servers run out of GPU memory.

What is stored

Inside each layer of the model, every token is turned into two vectors: a key, which says what the token can be matched on, and a value, which is what it contributes when matched. A new token is compared with the keys of all earlier tokens and collects their values. The keys and values of earlier tokens never change, so they are computed once and kept. That store, for every layer and every token so far, is the KV cache.

Without it, writing token 1,000 would mean processing all 999 earlier tokens again. With it, only the new token is processed.

How big it is

For one token the cache holds:

2 × layers × KV heads × head dimension × bytes per value

The 2 is for keys and values. For Llama 3 8B that is 2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes, or 128 KiB for every token. Multiply by the context length and by the number of conversations being served together:

Llama 3 8B, 16-bitCache
1 conversation, 2,048 tokens0.25 GiB
1 conversation, 8,192 tokens1 GiB
8 conversations, 8,192 tokens each8 GiB
32 conversations, 8,192 tokens each32 GiB

The weights of the same model take about 15 GiB and do not grow. At a batch of 16 the cache is already larger than the model.

Why batches multiply it

The weights are the model, and one copy serves every request. The cache belongs to one conversation: it holds what the model has worked out about that particular text. Eight conversations need eight caches. This is why a model that fits comfortably for a single user can fail to fit when a server tries to handle several at once.

Three ways it is made smaller

What it means for RAG

Retrieved chunks are part of the prompt, so they are part of the cache. Five chunks of 300 words add about 2,000 tokens to every request, which for Llama 3 8B is about 250 MiB of cache per conversation. Retrieving fewer or shorter chunks saves GPU memory as well as API cost. The chunk size guide covers how to choose.

Words you will meet