Speculative Decoding Simulator
See how a small draft model and one checking pass by the large model can produce several tokens at once, and when that is slower than plain decoding. Nothing is uploaded.
- 1Set the draft length and acceptance rate
- 2Run a few rounds
- 3Find the best draft length on the heatmap
Settings
A simplification: every token is accepted with the same probability, independently of the others. Real text is accepted in runs.
How it is worked out
Draft and verify
Speed against plain decoding
What it is for
- Understand how speculative decoding produces several tokens from one pass of the large model.
- See why the acceptance rate of the draft model matters more than anything else.
- Find the draft length where extra guesses stop paying for themselves.
- See the settings where speculation is slower than not using it.
How it works
A large language model writes one token per pass, and each pass is slow. Speculative decoding adds a much smaller draft model. The draft model quickly guesses the next few tokens. The large model, called the target, then checks all of those guesses in a single pass, which costs about the same as producing one token.
The target keeps the guesses it agrees with, up to the first one it rejects. There it puts its own token and the rest of the draft is thrown away. So a round always keeps at least one token and at best keeps the whole draft plus one more. The text that comes out is what the target would have written on its own.
How many tokens a round keeps depends on how often the draft is right. If each guess is accepted with probability α and the draft is k tokens long, a round keeps (1 − α^(k+1)) ÷ (1 − α) tokens on average. A round costs one target pass plus k draft passes, each a fraction c of a target pass. The speedup is the first number divided by the second.
This simulator assumes every token is accepted with the same probability, independently of the others. That is a simplification. In real text, easy stretches are accepted in long runs and hard ones are rejected again and again, and real systems also pay some overhead for running two models. Treat the numbers as a guide to the shape of the trade-off.
Questions
What is speculative decoding?
A way to speed up a large language model without changing its output. A small model drafts several tokens ahead, and the large model checks them all in one pass, keeping the ones it agrees with.
Does it change the answer?
No. With the standard acceptance rule the result follows the large model's own distribution. Only the speed changes.
What is the acceptance rate?
The share of draft tokens the target model agrees with. It depends on how good the draft model is and how predictable the text is. Code and boilerplate are accepted more often than open-ended writing.
Why is a longer draft not always better?
One wrong guess throws away everything after it. The chance that many guesses in a row are all right falls quickly, while the cost of making them is paid every time.
When is speculative decoding slower?
When the draft model is often wrong, or is not much cheaper than the target. Then most rounds pay for several draft passes and keep only one token.
What does the draft cost ratio mean?
The time one draft pass takes compared with one target pass. A draft model a tenth of the size is roughly 0.1.
Can I share my settings?
Yes. The settings are kept in the page address, so Copy link gives a link that opens the page exactly as you left it. CSV and JSON download the inputs and the results.
Is anything uploaded?
No. The page is a calculator that runs in your browser tab. The numbers you enter stay on your device.