How LLM inference works
By TryRAG · Updated
Try it: the inference playgrounds are seven pages to work through in order. Start with Prefill vs Decode.
Retrieval, the subject of most of this site, decides what goes into the prompt. Inference is what happens after that: a model running on a GPU reads the prompt and writes the answer. This page is the map. It follows one request and shows where each idea fits.
One request
A request has two phases. First the model reads the prompt, all at once. Then it writes the answer one token at a time: each new token needs one pass through the whole model, and each pass looks back at everything before it. Writing is the slow part, because a 300-token answer is 300 passes.
Two things sit in GPU memory while this happens. The weights are the model itself, loaded once and shared by every request. The KV cache is the model's stored working for one conversation, and every conversation being served needs its own.
The first playground, Prefill vs Decode, shows why these two phases are limited by different things: the first by the GPU's arithmetic, the second by how fast it can read the model from memory. The five ideas below build on that.
The five ideas, in order
- The KV cache decides what fits. It grows with the length of the prompt and answer, and it is multiplied by the number of conversations served at once. On a busy server it can be larger than the model. KV Cache Calculator · guide
- Quantization shrinks the weights. Storing each weight in 4 or 8 bits instead of 16 makes the model a quarter or half the size, at the price of a small rounding error. What it frees can hold more cache. Quantization Visualizer · guide
- Batching uses the room. A GPU can write the next token for many conversations in one pass, almost as fast as for one. How the server fills and refills that batch decides how many tokens per second it really gets. Batching Simulator · guide
- Speculative decoding speeds up one answer. A small model guesses several tokens ahead and the large model checks them in a single pass. This is a separate idea from batching: it makes each answer arrive sooner. Speculative Decoding Simulator · guide
- Cost follows from the rest. A GPU is paid for by the hour. Tokens per second, and the share of the time the GPU is actually busy, turn that into a cost per token that can be compared with an API price. Inference Cost Calculator · guide
Two more playgrounds sit alongside these: Prefill vs Decode (guide), which explains the two phases of a request, and the Prefix Caching Simulator (guide), which shows how much of a prompt a server can avoid reading twice.
How they connect
The five are one chain. Memory is fixed, so smaller weights (2) leave more room for cache (1). More room for cache means more conversations in a batch (3). A bigger batch, and more tokens per pass (4), mean more tokens per second. More tokens per second from the same hourly bill means a lower cost per token (5), as long as there is enough traffic to keep the GPU busy.
Each playground ends with a link to the next and carries your numbers across where they apply: the model and GPU you chose in the KV calculator open in the quantization page, and the speed-up you find in the speculative decoding page is applied in the cost calculator.
Where RAG comes in
Retrieved chunks are part of the prompt. A longer prompt takes longer to read, fills more cache for every conversation, leaves room for fewer conversations in a batch, and costs more on an API that charges for each prompt token. That is why chunk size and top-k matter for cost as well as for answer quality. Tick "Add retrieved chunks" on the KV cache, batching and cost pages to see the effect.
Words you will meet
- Inference: running a trained model to get an answer, as opposed to training it.
- Token: a short piece of text; see tokens and cost.
- Prefill: reading the prompt before the first token is written.
- Decoding: writing the answer, one token per pass.
- Throughput: tokens written per second across all requests.