One of the most elegant inference speedups in wide use costs nothing in hardware and nothing in quality: speculative decoding. It makes a large model generate 2–3x faster by letting a small one do the guessing.
The idea
Normally a large model generates one token at a time — each token a full, expensive forward pass. Speculative decoding adds a small, fast draft model that rapidly proposes several tokens ahead. The large model then verifies that whole guessed sequence in a single pass, accepting the tokens that match what it would have produced and correcting from the first mismatch.
Guessing is cheap; checking is cheap in bulk. Speculative decoding turns many slow steps into a few fast ones.
Why it's free quality
The crucial property: the output is identical to what the large model would have produced on its own. The draft model only proposes; the large model has final say on every token. So you get the exact same answer, just faster. The speedup comes entirely from doing fewer expensive forward passes, not from cutting corners.
Where it fits
Speculative decoding pairs naturally with quantization — one reduces the cost per computation, the other reduces the number of computations, and they compose. Together they're a big part of why serving costs keep falling. For anyone running their own inference, it's close to a free lunch: same output, meaningfully lower latency and cost. The main work is choosing or training a good draft model whose guesses the big model accepts often.
