Speculative decoding, explained
By TryRAG · Updated
Try it: the Speculative Decoding Simulator runs draft-and-verify rounds and maps the speedup for every draft length and acceptance rate.
A large language model writes one token per pass through the network, and each pass is slow. Speculative decoding makes it faster without changing what it writes. A small model guesses several tokens ahead, and the large model checks all the guesses in one pass.
The idea
Much of any text is easy to predict. After "The capital of France is" almost any model will say "Paris". Using the large model for such tokens is a waste. But nobody knows in advance which tokens are easy, so the small model guesses all of them and the large model decides which guesses to keep.
One round, step by step
- Draft. A small, fast model writes the next few tokens, say four.
- Verify. The large model, called the target, reads all four in a single pass. A model can check many positions at once in about the time it takes to write one token.
- Accept. The target keeps the draft tokens up to the first one it disagrees with.
- Correct. At that point it supplies its own token, and the rest of the draft is discarded. If it agreed with all four, it adds a fifth of its own.
Every round therefore produces between one and five tokens for the cost of one large pass and four small ones. The acceptance rule is designed so that the result follows the target model's own choices: the text is the same as if the target had written it alone.
How much faster
Call the draft length k, the chance that a draft token is accepted α, and the cost of one draft pass compared with one target pass c. If each token is accepted independently:
Tokens per round = (1 − αk+1) ÷ (1 − α)
Speedup ≈ tokens per round ÷ (1 + k × c)
| Draft is right | Tokens per round (k = 4) | Speedup (c = 0.1) |
|---|---|---|
| 90% of the time | 4.10 | 2.93× |
| 80% | 3.36 | 2.40× |
| 60% | 2.31 | 1.65× |
| 20% | 1.25 | 0.89×, slower |
When it does not help
- The draft model is often wrong. Most rounds then keep one token and still pay for all the guesses. Below a certain acceptance rate the result is slower than not speculating.
- The draft model is not much cheaper. A draft model half the size of the target leaves little to gain.
- The draft is too long. One wrong guess throws away everything after it, so the chance of using the tenth guess is small while its cost is certain. There is a best draft length, usually between 3 and 8.
- The server is already full. Speculation spends extra computation to save time. On a GPU already busy with a large batch, that computation is not spare.
Words you will meet
- Target model: the large model whose output you want.
- Draft model: the small model that guesses ahead. Also called the assistant model.
- Acceptance rate: the share of draft tokens the target keeps.
- Lookahead: another name for the draft length.