Speculative Prefill: Training-Free Token Importance for Faster TTFT

Jingyu Liu, Beidi Chen, Ce Zhang · ICML 2025 · A small model picks the tokens, the big model prefills only those
arXiv:2502.02789 Speaker: Llama-3.1-8B Base: Llama-3.1-70B / 405B (FP8) Training-free

Prompts are ~10× longer than outputs Paper-cited ratio · unit scale illustrative

TTFT is essentially the prefill wait. Hover the blocks.

Where the prefill cost sits Illustrative · block heights not measured

Sparse attention only touches attention. Dropping tokens shrinks MLP, attention and TP communication together.

Why TTFT cuts become QPS Illustrative · schematic, not measured

Paper argument: max QPS under a timeout scales roughly as 1/TTFT (an approximation that assumes a long timeout).

SpecPrefill, step by step Illustrative 24-token prompt

Scores and keep rate here are made up to show the mechanics. The paper fixes the keep rate per task.

From query rows to kept chunks Illustrative · synthetic scores

Max over heads and layers, mean over look-ahead steps, optional avg-pool, chunk mean, top chunks up to the keep rate. Row 0 is the last prompt token.

With no look-ahead, the sink and recency tokens can win. Averaging several decoded steps dilutes both biases. The paper reports little gain past 16 steps.

Results

Don't merge the headlines. 7.66× TTFT = TTFT measurement (405B FP8, 10% keep, 16 prompts × 32k). 7× QPS = separate end-to-end serving test (405B, Multi-Doc QA, 10% keep).

TTFT speedup ranges, as reported Reported

405B at 10% keep: ~6.7× (128 prompts × 4k) up to 7.66× (16 × 32k), roughly 7.7 s down to 1 s. The 70B with look-ahead gets ~2.6–3× against a 10× cut in tokens.

Where it sits among related methods Conceptual map

Click a node. Green = reduces prefill work. Grey = doesn't improve TTFT.

The two bets

Hover or click each bet.

Hover a bet.

Open edges

Keep rate fixed per task after seeing results No variance, seeds or CIs Speculator runs sequentially No per-token logits Multi-turn: dropped tokens have no KV Needs a compatible model family Score is an attention heuristic, not information-theoretic No 405B counting-task results

References

  1. Liu, Chen, Zhang. Speculative Prefill, 2025. arXiv:2502.02789
  2. Li et al. SnapKV, 2024. scholar
  3. Fu et al. LazyLLM, 2024. scholar
  4. Tang et al. (Quest); Liu et al. (RetrievalAttention), 2024. scholar
  5. Shi et al. GemFilter, 2024. scholar
  6. Qiao et al. SwiftKV, 2024. scholar
  7. Leviathan, Kalman, Matias. Speculative Decoding, 2023. scholar
  8. Jiang et al. LLMLingua-2 / LongLLMLingua, 2023-24. scholar
  9. Agrawal et al. (Sarathi-Serve); Qin et al. (Mooncake), 2024. scholar
  10. Liu et al. Lost in the Middle, 2023. scholar
  11. Pan et al. (Marconi); Yao et al. (CacheBlend), 2024-25. scholar