Draft-Verify Inference: One Parallel Pass, Five Candidate Tokens
Two ways to get k tokens per step.
- Left column - Speculative decoding (two models):
- Small draft proposes k tokens, one at a time
- Big target verifies all k in one pass
- Each position needs only the past: parallel
- Accept rule: keep matches, fix first mismatch
- Right column - Multi-token prediction (one model):
- Aux heads predict tokens +2, +3...
- Loss = next-token loss + aux losses
- At inference: heads propose, main verifies
- Changes architecture and training
- Why it works - one weight stream, k tokens:
- Decoding: every token streams all weights
- Verifying k costs about ONE weight stream
- So ~k tokens per step: 2-3x faster
- 20-40% is batch ~60-120 MFU, a band
- That % is MFU, not nvidia-smi (reads ~100%)
- The exactness rule - and how it breaks:
- Exact only under rejection sampling
- Keep-all-draft-tokens breaks the distribution
- Blind MTP heads = approximation
Sticky note - the whole lesson: One weight stream per step, k tokens from idle compute.