Prefix caching, explained
By TryRAG · Updated
Try it: the Prefix Caching Simulator splits a prompt into system prompt, chunks and question, shows what can be reused, and compares ways of routing requests across servers.
Reading a prompt costs time, and a great deal of it is the same from one request to the next. Prefix caching lets a server keep its work on the beginning of a prompt and skip it when the same beginning arrives again.
The idea
When a model reads a prompt it builds the keys and values for every token, the KV cache. That is the work of the prefill phase. If a later request starts with exactly the same tokens, the keys and values for those tokens are already there. The server reads only what is new, and the first token arrives sooner. Servers such as vLLM and SGLang can do this automatically.
Only a prefix can be reused
The reusable part must be identical, token for token, from the very first token. Each token's keys and values depend on everything before it, so if one earlier token differs, every token after it differs too. A date at the top of the prompt, a user's name in the first line, or a different order of two instructions can make the whole rest of the prompt unreusable.
What this means for a RAG prompt
| Part of the prompt | Same in every request? | Reusable? |
|---|---|---|
| System prompt and instructions | Yes | Yes, if it comes first |
| Retrieved chunks | Rarely: they depend on the question | Only if the same chunks come back, in the same order |
| The question | No | No |
The system prompt is a small part of a RAG prompt and the retrieved chunks are a large part. In the simulator's example, a 1,500-token system prompt, 2,000 tokens of chunks and a 40-token question, only about half of the prompt can be reused even when 10% of requests retrieve the same chunks. A hit cuts the wait for the first token from about 690 ms to about 360 ms on an RTX 4090. Caching helps RAG less than it helps a chat with a long fixed prompt.
Practical rules: put everything fixed first, then the retrieved chunks, then the question. Keep anything that changes, such as the date, out of the fixed part or at the end. Where the same chunks are retrieved often, as with a FAQ, keeping them in a stable order lets them be reused as well.
Memory as well as time
Requests running at the same time that share a prefix can share one copy of its cache instead of keeping one each. With 1,700 reusable tokens, 8 requests at once and Llama 3 8B, that is about 1.5 GiB of cache memory saved, room for more requests in the batch.
Routing across servers
A busy service runs many servers, and each has its own cache with room for only a few prefixes. If requests go to servers in turn (round-robin), every server sees every prefix and keeps pushing the older ones out. If requests with the same prefix always go to the same server, each server's few prefixes stay warm.
| Routing | Cache hit rate | Busiest server's share of requests |
|---|---|---|
| Round-robin | 46% | 25% (even) |
| By prefix | 98% | 37% |
| By prefix, with a load limit | 93% | 31% |
These are the simulator's starting settings: 4 servers, 12 different prefixes, room for 4 on each, and a few prefixes more popular than the rest. Routing by prefix more than doubles the hit rate, but it loads the servers unevenly, because a popular prefix makes its server busier. A load limit sends a request elsewhere when its server is far busier than average, which gives up a few hits to keep the load level. The simulation is a simplification: it ignores queueing and the cost of moving a cache between servers.
When it does not help
- Every prompt is different. With nothing in common at the start, there is nothing to reuse.
- Prompts are short. Prefill is already quick, so there is little to save.
- The cache is too small for the prefixes in use. Entries are dropped before they are used again. More prefixes than room is when routing matters most.
Words you will meet
- Cache hit / miss: the prefix was found / not found in the server's cache.
- Eviction: dropping a cached prefix to make room, usually the one used longest ago.
- Cold start: the first request with a prefix, which has to read it in full.
- Prefix-aware (or cache-aware) routing: choosing a server by the prefix of the request.
- Time to first token: the wait before the first word of the answer.