Prefix caching, explained

By TryRAG · Updated

Try it: the Prefix Caching Simulator splits a prompt into system prompt, chunks and question, shows what can be reused, and compares ways of routing requests across servers.

Reading a prompt costs time, and a great deal of it is the same from one request to the next. Prefix caching lets a server keep its work on the beginning of a prompt and skip it when the same beginning arrives again.

The idea

When a model reads a prompt it builds the keys and values for every token, the KV cache. That is the work of the prefill phase. If a later request starts with exactly the same tokens, the keys and values for those tokens are already there. The server reads only what is new, and the first token arrives sooner. Servers such as vLLM and SGLang can do this automatically.

Only a prefix can be reused

The reusable part must be identical, token for token, from the very first token. Each token's keys and values depend on everything before it, so if one earlier token differs, every token after it differs too. A date at the top of the prompt, a user's name in the first line, or a different order of two instructions can make the whole rest of the prompt unreusable.

What this means for a RAG prompt

Part of the promptSame in every request?Reusable?
System prompt and instructionsYesYes, if it comes first
Retrieved chunksRarely: they depend on the questionOnly if the same chunks come back, in the same order
The questionNoNo

The system prompt is a small part of a RAG prompt and the retrieved chunks are a large part. In the simulator's example, a 1,500-token system prompt, 2,000 tokens of chunks and a 40-token question, only about half of the prompt can be reused even when 10% of requests retrieve the same chunks. A hit cuts the wait for the first token from about 690 ms to about 360 ms on an RTX 4090. Caching helps RAG less than it helps a chat with a long fixed prompt.

Practical rules: put everything fixed first, then the retrieved chunks, then the question. Keep anything that changes, such as the date, out of the fixed part or at the end. Where the same chunks are retrieved often, as with a FAQ, keeping them in a stable order lets them be reused as well.

Memory as well as time

Requests running at the same time that share a prefix can share one copy of its cache instead of keeping one each. With 1,700 reusable tokens, 8 requests at once and Llama 3 8B, that is about 1.5 GiB of cache memory saved, room for more requests in the batch.

Routing across servers

A busy service runs many servers, and each has its own cache with room for only a few prefixes. If requests go to servers in turn (round-robin), every server sees every prefix and keeps pushing the older ones out. If requests with the same prefix always go to the same server, each server's few prefixes stay warm.

RoutingCache hit rateBusiest server's share of requests
Round-robin46%25% (even)
By prefix98%37%
By prefix, with a load limit93%31%

These are the simulator's starting settings: 4 servers, 12 different prefixes, room for 4 on each, and a few prefixes more popular than the rest. Routing by prefix more than doubles the hit rate, but it loads the servers unevenly, because a popular prefix makes its server busier. A load limit sends a request elsewhere when its server is far busier than average, which gives up a few hits to keep the load level. The simulation is a simplification: it ignores queueing and the cost of moving a cache between servers.

When it does not help

Words you will meet