Prefix Caching Simulator

See how much of a prompt a cache can reuse, why retrieved chunks help it less than the system prompt, and why servers route requests by prefix. Nothing is uploaded.

Model and GPU

The prompt

Servers and routing

Where the numbers come from

Estimates. First-token times come from the model on the Prefill vs Decode page. Routing is simulated on 600 requests.

How much of the prompt can be reused

Three ways to route requests

PolicyCache hit rateAverage first tokenBusiest server

Hit rate as the number of prefixes grows

How it is worked out

    What it is for

    How it works

    When a model reads a prompt it builds the keys and values for every token and stores them in the KV cache. If the next request starts with exactly the same tokens, a server that kept that cache can skip the work for those tokens and read only what is new. This is prefix caching. It makes the first token arrive sooner and lets requests share one copy of the cache in memory.

    Only a prefix can be reused, meaning a run of tokens from the start that is identical, token for token. The system prompt is the standard case: every request begins with it. In a RAG prompt, the retrieved chunks come next, and they usually differ from one question to the next, so they are not reused. If one token earlier in the prompt differs, everything after it must be read again. This is why fixed instructions belong first and the retrieved chunks after them.

    Large systems run many servers, and each one has its own cache with room for only a few prefixes. If requests go to servers in turn (round-robin), every server sees every prefix and keeps pushing older ones out, so the hit rate falls as the number of prefixes grows. Sending requests with the same prefix to the same server keeps each server's few prefixes warm. The price is balance: a popular prefix makes its server busier than the others. A load limit, which sends a request elsewhere when its server is far busier than average, gives up some hits to keep the servers even.

    This simulator splits a prompt into the system prompt, retrieved chunks and the question, and estimates the first-token time from the prefill model of the Prefill vs Decode page. The routing part sends 600 requests, chosen with a fixed random seed so reloads give the same result, to servers that each keep their most recently used prefixes. It leaves out queueing, cache eviction by memory pressure, and the cost of moving a cache between servers. Treat it as a guide to the direction and size of the effects.

    Questions

    What is prefix caching?

    Keeping the KV cache of a prompt's beginning so that a later request with the same beginning skips reading it again. The model only reads the part that is new.

    Why are retrieved chunks reused less than the system prompt?

    The system prompt is identical in every request. The retrieved chunks depend on the question, so two requests rarely share them. And a prefix must match from the very first token: one different token before the chunks and nothing after it can be reused.

    Does the order of the prompt matter?

    Yes. Put everything that stays the same first, then the retrieved chunks, then the question. A changing line near the top, such as the date or the user's name, spoils the cache for the whole prompt after it.

    What is routing by prefix?

    Choosing the server for a request by looking at its prefix, so requests that share a prefix reach a server that has already cached it. Round-robin ignores the prefix and spreads them evenly.

    What is the catch with routing by prefix?

    Uneven load. If one prefix is far more popular than the rest, the server that holds it gets more requests than the others. A load limit sends some of them elsewhere, at the cost of some cache hits.

    Does a cache hit change the answer?

    No. The cached keys and values are the same ones the server would compute again. Only the time to the first token changes.

    What does "cache memory saved" mean?

    When several requests running at once share a prefix, a server can hold one copy of its cache instead of one per request. The figure is the memory freed by sharing, for the requests running at the same time.

    Can I share my settings?

    Yes. The settings are kept in the page address, so Copy link gives a link that opens the page exactly as you left it. CSV and JSON download the inputs and the results.

    Is anything uploaded?

    No. The page is a calculator that runs in your browser tab. The numbers you enter stay on your device.