Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context
Speculative decoding accelerates autoregressive generation by having a cheap draft propose tokens that a target verifies in parallel. Frontier models increasingly ship a built-in Multi-Token-Prediction (MTP/NEXTN) draft head under the assumption that the draft is negligibly cheap. At million-token context this breaks: an MTP draft head typically runs full attention over the entire KV cache at every draft step, so its read grows linearly with context and comes to dominate the draft cost -- precisely where speculation is most valuable. The effect compounds with draft length (a deep native draft
Lineage graph
Paper → model → repo connections mined from source citations (Tier-1 exact match).
Why these links exist
Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.
- PossiblePossibly related (embedding) · 53%[Research] JetSpec: Speculative Decoding with Parallel Tree Drafting Enables up to 9.64x Lossless LLM Inference Speedup with more than 1000TPS →
- PossiblePossibly related (embedding) · 52%I benchmarked 13 models at 65K-128K context to find out what actually matters for agentic workloads →
- PossiblePossibly related (embedding) · 52%New Server Hopes to Break Through AI’s “Memory Wall” →
- LinkedLinked via arxiv author · 85%Alagappan Valliappan →
“Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context”