1
00:00:01,000 --> 00:00:07,919
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:07,919 --> 00:00:45,350
[Hal Turing] Today's paper is "Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models," by Ekagra Ranjan and colleagues Komal Teru, Bharat Venkitesh and Acyr Locatelli, all at Cohere, from 2026. They measured ten sparse mixture-of-experts models, from 30 billion to a trillion parameters. The accepted wisdom was that speculative decoding barely helps these models when you serve one request at a time. They find that for many of the newer ones, a single request is where it helps most.

3
00:00:45,350 --> 00:01:04,344
[Dr. Ada Shannon] That matters because single-request serving is where a lot of real traffic sits now: agents, test-time scaling, and the long tail of reinforcement-learning rollouts, where a few slow generations hold everything up. If the standard cost model says don't bother at batch one, and it's wrong, people are leaving latency on the table.

4
00:01:04,344 --> 00:01:08,848
[Hal Turing] Let's build up to why anyone believed that. Start with plain speculative decoding.

5
00:01:08,848 --> 00:01:49,901
[Dr. Ada Shannon] Decoding one token at a time is memory-bandwidth-bound at small batch. The GPU spends its time streaming weights from memory and does very little arithmetic. Speculative decoding exploits that. A cheap drafter proposes K tokens. The big target model scores all K plus one positions in a single forward pass, and rejection sampling keeps a prefix of the drafts. The output distribution is provably identical to the target's. That's the Leviathan paper from Google in 2022 and the DeepMind speculative sampling paper by Chen and colleagues in 2023. For a dense model, the weights are read once no matter how many positions you score, so the extra tokens are nearly free.

6
00:01:49,901 --> 00:01:51,387
[Hal Turing] And an MoE breaks that how?

7
00:01:51,387 --> 00:02:11,589
[Dr. Ada Shannon] In a mixture-of-experts layer there are N expert sub-networks, and a router picks the top k for each token. One token reads a small slice of the experts. But K plus one tokens can route to different experts, so the verification pass may have to read many more distinct expert weights. The free-extra-tokens argument stops being obviously true.

8
00:02:11,589 --> 00:02:14,096
[Hal Turing] So sparsity ratio and expert size both matter.

9
00:02:14,096 --> 00:02:43,028
[Dr. Ada Shannon] Yes. Mixtral picks 2 of 8 experts, a ratio of 0.25, and each expert is about 168 megabytes in FP8. That's a coarse MoE. Qwen3-30B picks 8 of 128, with experts around 4.5 megabytes. Kimi-K2.5 picks 8 of 384. Qwen3-Next-80B picks 10 of 512, a ratio of about 0.02. Those are fine-grained: many small experts, so each token touches a tiny fraction of the layer.

10
00:02:43,028 --> 00:02:44,607
[Hal Turing] What did the prior work predict?

11
00:02:44,607 --> 00:03:25,800
[Dr. Ada Shannon] The closest is MoESD, by Zongle Huang and colleagues, NeurIPS 2025. It models verification cost as unique experts loaded times bytes per expert, assuming a bandwidth-bound regime. That predicts a concave curve of speedup against batch size. Gains are limited at batch one, peak at an intermediate batch where expert activation saturates, then fall. Concave here just means up, peak, down. The source paper says MoESD's view matched that shape on the MoEs it studied. It doesn't list the full set. My briefing says they were mostly coarse models like Mixtral and Qwen2-57B, so treat that as secondhand.

12
00:03:25,800 --> 00:03:29,886
[Hal Turing] So is this paper contradicting MoESD, or looking at a different regime?

13
00:03:29,886 --> 00:03:36,806
[Dr. Ada Shannon] That's the question to hold onto. The two papers may mostly cover different kinds of models. We'll come back to it in the last part.

14
00:03:36,806 --> 00:03:38,803
[Hal Turing] Are there other papers it argues with?

15
00:03:38,803 --> 00:04:06,667
[Dr. Ada Shannon] Two. Utility-driven speculative decoding, by Saxena, Tsai, Taneja, Jaleel and Qureshi at MICRO 2025, is cited as saying speculative decoding is weak for MoE. Its practical answer is to adapt per request using a utility estimate. And MagicDec, by Sadhukhan and colleagues at ICLR 2025, shows speculative decoding helps at large batch when KV-cache traffic dominates. We'll unpack those later.

16
00:04:06,667 --> 00:04:10,707
[Hal Turing] Before the results, give me the metrics, so I know which direction is good.

17
00:04:10,707 --> 00:04:39,082
[Dr. Ada Shannon] The headline metric is TPOT, time per output token. Speedup is vanilla TPOT divided by speculative TPOT, so higher is better. The second metric is target efficiency, or TE. It's the time of one ordinary target step divided by the time of the K plus one verification pass. TE removes draft acceptance from the picture and isolates verification cost. A TE of 1.0 would mean verification is free. Higher is better.

18
00:04:39,082 --> 00:04:42,100
[Hal Turing] Why bother with a second metric if speedup is what users feel?

19
00:04:42,100 --> 00:05:04,159
[Dr. Ada Shannon] Because speedup mixes two things. One is how often drafts get accepted, which depends on the drafter and the text. The other is how expensive verification is, which depends on the target model, kernels and hardware. The paper is about the second one, so TE is the cleaner probe. It doesn't fully separate them, because the models use different drafting methods, and we'll return to that.

20
00:05:04,159 --> 00:05:06,295
[Hal Turing] Give me the preview, no numbers yet.

21
00:05:06,295 --> 00:05:40,197
[Dr. Ada Shannon] The curve is concave for Mixtral-style models, as predicted. For the newer fine-grained models it's non-concave, meaning speedup is highest at or near batch one. The authors propose two missing ingredients in the bytes model. One is temporal routing overlap: adjacent tokens tend to pick similar experts, so the union is smaller than independence predicts. The other is GPU execution dynamics: how much bandwidth the kernels actually achieve, and what the cache absorbs. Bytes alone isn't time.

22
00:05:40,197 --> 00:05:44,562
[Hal Turing] So the roofline intuition, bytes moved equals time, is the thing under audit.

23
00:05:44,562 --> 00:05:46,745
[Dr. Ada Shannon] Exactly. Next, how they measured it.

24
00:05:46,745 --> 00:05:48,370
[Hal Turing] Okay, what sits under the claim?

25
00:05:48,370 --> 00:06:21,575
[Dr. Ada Shannon] Ten models, from 30 billion to a trillion parameters. They all run in vLLM version 1 at a single commit, with three draft tokens, on MT-Bench. Outputs are fixed at 256 tokens and prefix caching is off. Everything runs on H100s except Kimi-K2.5 and DeepSeek-V3, which are on B200s. Each speedup was run three times, with variation under 5 percent at every batch size. The speculation method also differs by model: EAGLE, EAGLE-3, built-in multi-token-prediction heads, or a separate small draft model.

26
00:06:21,575 --> 00:06:22,596
[Hal Turing] And the headline?

27
00:06:22,596 --> 00:06:51,807
[Dr. Ada Shannon] The curve shape splits by model class. Mixtral and Qwen2-57B are concave, as predicted. They want a crowd before speculation pays off. The fine-grained models, like Qwen3-30B, Qwen3-235B and Kimi, peak at or near batch size 1. Target efficiency at batch 1 tracks the shape. One exception is Qwen3.5-35B, which is non-concave even though its efficiency is lower than GLM-4.7-Flash's.

28
00:06:51,807 --> 00:06:54,500
[Hal Turing] So what does the bytes model say for Qwen3-30B?

29
00:06:54,500 --> 00:07:18,556
[Dr. Ada Shannon] It says verifying four tokens costs about twice a normal step. Target efficiency of 0.51. They measured 0.82, which is about 1.2 times. The important detail is that the prediction already uses the real unique-expert count from actual routing. So the miss of 0.31 isn't a routing-count error. For Mixtral the miss is only 0.06.

30
00:07:18,556 --> 00:07:22,086
[Hal Turing] Then where does the saving come from? Routing is already counted.

31
00:07:22,086 --> 00:07:58,541
[Dr. Ada Shannon] Two places. The first is temporal overlap, which is already inside that prediction. They compare three baselines. Uniform assumes every expert is equally likely. Independence uses real expert popularity but treats tokens as unrelated. Empirical is measured on consecutive tokens. Adjacent tokens share 20 to 45 percent of their experts in the fine-grained models. Independence predicts 4 to 12 percent. For Qwen3-30B, verifying four tokens touches about 20 distinct experts per layer, not the roughly 29 that uniform routing implies. That's a third fewer.

32
00:07:58,541 --> 00:08:03,835
[Hal Turing] Is that overlap just an artifact of accepted tokens? Rejected drafts could route elsewhere.

33
00:08:03,835 --> 00:08:28,773
[Dr. Ada Shannon] They checked. Rejected tokens overlap with the preceding token about as much as accepted ones. So accepted-token overlap is a conservative proxy. The temporal part is 69 to 92 percent of total overlap in six sparse models, but only about 29 to 32 percent in Mixtral and Qwen2-57B. Those two have a large combinatorial baseline, so there's little room left.

34
00:08:28,773 --> 00:08:33,046
[Hal Turing] How do you turn overlap statistics into an expert count without running the model?

35
00:08:33,046 --> 00:09:17,535
[Dr. Ada Shannon] Iterative proportional fitting. For one expert, take its on/off state across the four verification tokens. That gives 16 joint states. Constrain the distribution to match the expert's measured activation probability and the pairwise overlap at gaps of one, two and three tokens. Of all distributions that fit, pick the maximum-entropy one, which makes the fewest extra assumptions. Then the chance the expert is used at least once is one minus the chance of no activation. Sum over experts and you have the expected unique count. The uniform variant assumes equal expert probabilities. The skewed variant uses real per-layer probabilities. Skewed is within 4 percent on every measured model.

36
00:09:17,535 --> 00:09:20,090
[Hal Turing] That covers routing. What's left of the 0.31?

37
00:09:20,090 --> 00:09:48,650
[Dr. Ada Shannon] The GPU. For Qwen3-30B on H100, the expert layers are two thirds of the weights but only a third of runtime. At batch 1 the attention kernels launch only 16 thread blocks across 132 streaming multiprocessors. Most of the chip sits idle, so they reach 22 to 25 percent of peak memory bandwidth. The expert kernels reach 45 to 49 percent. Bytes aren't time when bandwidth utilization differs by about two times.

38
00:09:48,650 --> 00:09:49,672
[Hal Turing] And the cache?

39
00:09:49,672 --> 00:10:29,889
[Dr. Ada Shannon] The default H100 kernel gives each token-expert pair its own thread block. When several verification tokens hit the same small expert, the repeat reads come from the 50 MB L2 cache. The hit rate goes from 5.72 percent in normal decoding to 27.79 percent under verification. Mixtral's kernel groups tokens by expert, so there are no repeat reads. Its 168 MB experts wouldn't fit in L2 anyway. That fits the TE trend. For Qwen3-30B, going from bf16 to FP8 and from one GPU to two raises TE from 0.63 to 0.86, because smaller shards stay cached.

40
00:10:29,889 --> 00:10:32,536
[Hal Turing] Can they split the 0.31 among those causes?

41
00:10:32,536 --> 00:11:10,570
[Dr. Ada Shannon] Partly. They give a three-step correction from 0.511 to 0.872. Using the real MoE time share gets it to 0.679, which is 44 percent of the gap. Using the measured MoE scaling gets it to 0.843, another 43 percent. Accounting for non-MoE amortization finishes it, at 8 percent. The middle step lumps L2 reuse together with dilution of fixed pipeline overhead, so the L2 share alone isn't isolated. It's also a decomposition of one model on one GPU. It isn't a counterfactual ablation.

42
00:11:10,570 --> 00:11:13,449
[Hal Turing] Does the IPF routing hold up when you run it on hardware?

43
00:11:13,449 --> 00:11:37,459
[Dr. Ada Shannon] They replace the learned router with IPF-generated assignments on dummy weights, then run the real kernels. Uniform random routing underestimates target efficiency at small batch, and the error is bigger for sparser models. IPF routing tracks the real curves much more closely at small batch. Residual gaps remain, since only pairwise statistics are matched. Still, you can estimate verification cost for a design before training it.

44
00:11:37,459 --> 00:11:40,385
[Hal Turing] Two side results. One is mixture-of-experts drafters.

45
00:11:40,385 --> 00:12:06,391
[Dr. Ada Shannon] That's a design study only. At low batch an MoE drafter doesn't improve target efficiency. Any gain would have to come from longer acceptance if trained well, and they didn't train one. The second result: routing is a weak signal for rejecting drafts early. The largest effect, Cohen's d about minus 0.3, still leaves accepted and rejected distributions overlapping by roughly 85 percent.

46
00:12:06,391 --> 00:12:15,865
[Hal Turing] Okay, now the critique. The headline says the concave curve often doesn't hold. How much of that is the models, and how much is the test setup?

47
00:12:15,865 --> 00:12:51,856
[Dr. Ada Shannon] Some of it is setup, and the paper can't fully separate the two. The main sweep uses three draft tokens, MT-Bench prompts, fixed 256-token outputs, prefix caching off, and vLLM at one commit. Larger draft lengths were run on only three models, and the curve family didn't change. Larger K mostly pushed the curves down. Long context was run on one model, Qwen3-30B, with roughly 32k-token prompts and batches up to 24. Verification cost was better there at every batch size, still best at batch one, with an unexplained dip at batch four.

48
00:12:51,856 --> 00:12:53,109
[Hal Turing] And other datasets?

49
00:12:53,109 --> 00:13:21,298
[Dr. Ada Shannon] Four datasets, but only for two Qwen3 models. That speaks to acceptance effects. It doesn't speak to kernels, because target efficiency is deliberately acceptance-independent. The bigger gap is the framework. Only vLLM was tested. SGLang and TensorRT-LLM were not tested, and the source gives no evidence about them. The test that would settle it is cheap: rerun the batch-one target efficiency measurement on fused-MoE kernels from those stacks.

50
00:13:21,298 --> 00:13:31,794
[Hal Turing] The drafters also differ. EAGLE, EAGLE-3, built-in multi-token heads, a small draft model. Doesn't that contaminate the curve shape?

51
00:13:31,794 --> 00:13:44,426
[Dr. Ada Shannon] For end-to-end speedup, yes. Acceptance and draft latency vary by model. Target efficiency reduces that confound but doesn't remove it. That's why I'd trust the verification-cost claim more than the exact speedup peaks.

52
00:13:44,426 --> 00:13:50,277
[Hal Turing] Now the mechanism. If the L2 effect comes from one kernel's behavior, what survives with a better kernel?

53
00:13:50,277 --> 00:14:25,571
[Dr. Ada Shannon] That's the central open question, and the paper never runs the ablation. Move to B200 and the L2 hit rate stays flat, about 2.5 to 2.2 percent, because the default kernel there groups tokens by expert. Yet target efficiency is still 0.76 against 0.51 predicted. So a residual advantage survives without L2. The authors call fixed per-step overheads the likely cause, and that is a hypothesis. Running the H100 kernel on B200 brings the hit rate back to about 27 percent, which says L2 behavior follows the kernel, not the chip.

54
00:14:25,571 --> 00:14:29,844
[Hal Turing] So the surprise is real, but the H100 explanation is one version of it.

55
00:14:29,844 --> 00:14:47,305
[Dr. Ada Shannon] Right. And the correction chain is a decomposition of one model on one GPU, not a controlled experiment. Nobody can say how much of the gap is L2 alone. Also, the bytes baseline already contains temporal overlap, so the 0.31 gap isn't an overlap effect.

56
00:14:47,305 --> 00:15:02,630
[Hal Turing] The attention side bothers me too. The bandwidth gap comes from attention kernels using 16 of 132 SMs at batch one. Is that a property of sparse MoEs, or of attention kernels that aren't tuned for that regime?

57
00:15:02,630 --> 00:15:23,575
[Dr. Ada Shannon] The source doesn't test better-tuned attention, so it can't answer. If attention used the chip better, MoE's share of runtime would rise and the effect would likely shrink. It's a fair challenge. But default stacks run these attention kernels at batch one today, so the finding describes real deployments even if it isn't permanent.

58
00:15:23,575 --> 00:15:25,665
[Hal Turing] Does this actually contradict MoESD?

59
00:15:25,665 --> 00:15:51,857
[Dr. Ada Shannon] Mostly no. MoESD's view matched concave curves on the MoEs it studied, and this paper agrees for Mixtral and Qwen2-57B, where the bytes gap is small. The disagreement is in the fine-grained regime, plus a hardware term the bytes view lacks. So it's mostly different regimes, not a pure contradiction. I wouldn't claim more about what MoESD tested than the briefing and the source support.

60
00:15:51,857 --> 00:15:56,222
[Hal Turing] What about the utility-driven work, which warns SD is weak for MoE?

61
00:15:56,222 --> 00:16:30,216
[Dr. Ada Shannon] Their remedy is per-request adaptation from a utility estimate. This paper says SD just works at batch one for fine-grained models. Those can both hold, since this is a model-class result. A serving stack might still want a utility check for coarse MoEs or higher batch. The source never compares the two empirically. MagicDec is the dense counterpart: long-context KV reads dilute verification cost. The long-context result here fits that pattern, but dense KV-dominated and MoE weight-dominated are different regimes.

62
00:16:30,216 --> 00:16:32,166
[Hal Turing] What should practitioners do differently?

63
00:16:32,166 --> 00:16:49,721
[Dr. Ada Shannon] Measure verification cost on your own hardware and kernel instead of deriving it from weight bytes. The synthetic-routing method gives a train-free screen for sparser targets and for MoE versus dense drafters. But the MoE-drafter and 896-expert results are design studies with no trained acceptance lengths.

64
00:16:49,721 --> 00:16:51,393
[Hal Turing] And the limits the authors admit?

65
00:16:51,393 --> 00:17:14,427
[Dr. Ada Shannon] The routing model fits pairwise overlap only and ignores higher-order and position effects. Whether overlap statistics transfer across model families is open, and screening a sparser model needs those statistics. Future directions include why some architectures have high temporal overlap, other serving frameworks, a kernel ablation, and trained MoE drafters.

66
00:17:14,427 --> 00:17:15,727
[Hal Turing] So where does that leave us?

67
00:17:15,727 --> 00:17:43,127
[Dr. Ada Shannon] The bytes view isn't a safe cost model for fine-grained MoE at batch one. Under default vLLM kernels on H100 and B200, many of these models gain most from speculation when run alone. Temporal expert overlap is real and predictable. The boundary: one framework, mostly three draft tokens, short MT-Bench outputs, a kernel-specific L2 story, no clean split of the gap, and no head-to-head with utility-driven speculation.

68
00:17:43,127 --> 00:17:48,560
[Hal Turing] Short version: check your own kernel before you trust the bytes. Thanks for listening, everyone.