AI Post Transformers · Episode Companion

Making Every Verified Token Count in MoE Speculative Decoding

arXiv:2605.00342 Lehan Pan, Ziyang Tao, Ruoyu Pang, et al. · 2026 USTC · Tianyijiaotong Technology Ltd. 79–89% of decode time is verification

Speculative decoding was sold on "verification is nearly free." This episode traces the lineage from Leviathan's original trick through Medusa and EAGLE-3, then shows why sparse Mixture-of-Experts targets break that assumption — and how EVICT prices tree size against real expert-loading cost to claw the speedup back, losslessly.

From One Guess to a Priced Tree lineage

Click a stage to see what it changed. Each step builds directly on the one before it.

Select a stage above.

Why Verification Used to Be "Free" dense vs MoE

Same tree, same verification pass — but a dense target loads one weight set no matter how many candidates it checks. An MoE target loads a different expert set per branch.

Shared dense FFN (loaded once) Expert loaded by one branch Expert loaded by multiple branches (still paid once, but the union keeps growing)

One Draft Tree, Many Experts Figure 1 concept

Hover a leaf. Every candidate token at every depth can route to a different top-k expert set — verifying the tree means loading the union of all of them.

Tree Size vs. Cost Fig. 1

Past ~30 nodes both curves climb steadily on Qwen3-30B-A3B.

Verification's Share of Latency Fig. 2

Across three MoE models, checking guesses costs more than making them.

How EVICT Prices a Tree algorithm

Four steps, no hand-tuned knob. Click each to expand.

Select a step above.

Utility Curve U(k) live-computed

Expected accepted length over verification cost, maximized to pick the tree size k.

C(k): Profiled Once, Looked Up Forever offline table

Verification cost per tree size, measured once at model init — zero live timing at runtime.

Speedup by Model 6 benchmarks · 2 temperatures

Only Qwen3-30B-A3B reports a full autoregressive baseline; the other two are reported relative to EAGLE-3.

"Verify Less" vs. "Verify What's Worth It" EVICT / EAGLE-3

EVICT accepts fewer tokens per step — but cuts cost by far more.

Ablation: Confidence Threshold vs. Utility ρ sweep

ρ=1.0 is literally EAGLE-3 (keep everything). Every fixed threshold underperforms EVICT's adaptive utility.

Who Didn't Make It Into Table 1 evaluation gap

The two closest relatives in EVICT's own related-work section never appear in the head-to-head comparison.

Does the Bottleneck Survive Real Batching? batch size = 1 caveat

All headline numbers are single-stream. Under continuous batching, concurrent requests already touch most experts — the marginal cost of one tree's extra nodes shrinks toward zero.