1 00:00:01,000 --> 00:00:07,130 [Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon. 2 00:00:07,130 --> 00:00:47,068 [Hal Turing] Today's paper is "Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding." The first author is Yongan Fu, mostly at NVIDIA, with co-authors from Georgia Tech, HKU, the University of Chicago and MIT. It's on arXiv from July 2026. The claim is that one 8B checkpoint can run as a normal model, as a parallel diffusion decoder, or as its own speculative drafter. The abstract says about six times more tokens per forward pass than Qwen3-8B, and about four times the throughput. 3 00:00:47,068 --> 00:01:05,644 [Dr. Ada Shannon] Those are two different measurements, and the abstract puts them side by side. Keep that in mind for the whole episode. The paper asks three questions. Can autoregressive and diffusion training be harmonized in one model? Can diffusion accelerate decoding better than multi-token prediction? And does diffusion have long-term headroom? 4 00:01:05,644 --> 00:01:09,963 [Hal Turing] Before any of that, I need the problem. Why is normal decoding slow enough to bother? 5 00:01:09,963 --> 00:01:44,421 [Dr. Ada Shannon] Autoregressive decoding emits one token per forward pass, and each token sees only earlier ones. Every pass reloads the model weights and the KV cache from memory to produce a single token. At batch size one, the arithmetic units mostly sit idle waiting on memory. The decode step is memory-bound, not compute-bound. With many users batched together, the weight loads are shared and the hardware fills up. That's why cloud serving is efficient and a single-user local setup isn't. Everything in this paper tries to spend that idle compute on producing more than one token per pass. 6 00:01:44,421 --> 00:01:48,833 [Hal Turing] So diffusion language models are one route. How do they produce several tokens at once? 7 00:01:48,833 --> 00:02:09,917 [Dr. Ada Shannon] A masked diffusion model starts with a block of mask tokens and fills in several per pass. Attention is bidirectional, so positions in the block see each other. Examples are LLaDA, Dream and SDAR. The known problem is that earlier models like these tend to trail autoregressive models in accuracy. Bidirectional attention also makes KV caching awkward. 8 00:02:09,917 --> 00:02:12,610 [Hal Turing] And block diffusion is the patch for the caching part? 9 00:02:12,610 --> 00:02:34,994 [Dr. Ada Shannon] Partly. Block Diffusion, the BD3-LM paper by Marianne Arriola and colleagues, 2025, generates block by block. Attention is causal across blocks and bidirectional inside one. Finished blocks can reuse the KV cache, and block size becomes a dial between autoregressive and diffusion behavior. Nemotron builds on that layout. 10 00:02:34,994 --> 00:02:39,592 [Hal Turing] The other way to get several tokens per pass is speculative decoding. How does that work? 11 00:02:39,592 --> 00:03:16,326 [Dr. Ada Shannon] The paper is Fast Inference from Transformers via Speculative Decoding, by Yaniv Leviathan and colleagues at Google, ICML 2023. A cheap drafter proposes k tokens. The big model checks all of them in one parallel pass. You keep the longest prefix that agrees, plus one corrected token from the verifier. Leviathan's accept-or-reject rule makes sampled output follow exactly the target model's distribution. The average number of tokens kept per verify cycle is the acceptance length. This paper's main verification test is a greedy match, which is simpler than the sampling rule. Keep that distinction too. 12 00:03:16,326 --> 00:03:19,066 [Hal Turing] What's the incumbent drafter, then? The thing they have to beat? 13 00:03:19,066 --> 00:03:51,481 [Dr. Ada Shannon] Multi-token prediction heads, and Eagle3. Gloeckle and colleagues at Meta, 2024, trained extra output heads to predict future tokens. DeepSeek-V3 ships a similar module. Eagle3, by Yuhui Li and colleagues, 2025, is a small drafter fed the target model's internal features, and it's the baseline here. The limits are structural. The drafter is small, and it proposes tokens recursively, one after another. So reliability drops past a short horizon, and the embedding and LM head add their own cost. 14 00:03:51,481 --> 00:03:55,707 [Hal Turing] So what's the move in Nemotron-Labs-Diffusion? Skip the separate drafter? 15 00:03:55,707 --> 00:04:19,345 [Dr. Ada Shannon] Yes. The three modes mostly differ in the attention mask and the decoding loop. The same weights run with a causal mask for normal decoding and for verification, and with a block-bidirectional mask for diffusion. In self-speculation mode, the diffusion pathway drafts a block of tokens and the causal pathway verifies them. There's no second network to deploy and keep in sync. 16 00:04:19,345 --> 00:04:22,363 [Hal Turing] Now the numbers. You said the headline mixes metrics. Which ones? 17 00:04:22,363 --> 00:04:53,896 [Dr. Ada Shannon] Three. Tokens per forward, or TPF, is how many tokens get committed per model forward pass. Acceptance rate here is how many drafted tokens survive verification per cycle. Throughput is tokens per second on real hardware, and it depends on kernels, GPU and concurrency. They're related but not interchangeable. A linear self-speculation cycle costs two forward passes, one diffusion draft and one autoregressive verify. The paper itself says real TPF is the acceptance rate divided by two. 18 00:04:53,896 --> 00:04:57,425 [Hal Turing] So a raw acceptance count could look twice as good as the per-pass reality. 19 00:04:57,425 --> 00:05:22,642 [Dr. Ada Shannon] Right, if you compare it against a one-pass baseline without noticing. I won't give a verdict yet. The claim as stated is six times the tokens per forward pass of Qwen3-8B, and four times throughput on SPEED-Bench with SGLang on a GB200. Then there's a speed-of-light analysis, an oracle ceiling on how much parallelism diffusion mode could reach. It's an upper bound, not a deployable sampler. 20 00:05:22,642 --> 00:05:29,794 [Hal Turing] Good. So how does one set of weights learn all three jobs? I assume it isn't just training on both objectives and hoping. 21 00:05:29,794 --> 00:05:53,896 [Dr. Ada Shannon] It's one forward-backward pass with two views of each sequence. A noisy stream has some block of tokens masked, and a clean stream has the original text. The noisy stream is bidirectional inside a block and causal across blocks. The new part is that the clean stream is strictly causal. That lets the model compute the normal next-token loss on the clean stream, and the diffusion loss on the noisy stream, with no label leakage. 22 00:05:53,896 --> 00:05:56,683 [Hal Turing] And the earlier block diffusion work didn't do that? 23 00:05:56,683 --> 00:06:14,655 [Dr. Ada Shannon] In BD3-LM, the clean stream can look at future tokens within a block. That's fine for a pure diffusion model. It means you can't also train a clean left-to-right objective on that stream, because the model would see the answer. Nemotron's change makes AR and diffusion share the same pass. 24 00:06:14,655 --> 00:06:16,698 [Hal Turing] What does the combined loss look like? 25 00:06:16,698 --> 00:06:34,206 [Dr. Ada Shannon] AR loss plus 0.3 times the diffusion loss. The two losses have different magnitudes, so the weight keeps one from dominating the other. They sweep the weight, and 0.3 gives the best average for both diffusion mode and AR mode, though only at 8B and 25 billion tokens. 26 00:06:34,206 --> 00:06:38,061 [Hal Turing] The diffusion loss has that one-over-t weighting, right? Why does averaging matter? 27 00:06:38,061 --> 00:06:56,869 [Dr. Ada Shannon] At small noise levels, only a few tokens are masked, but each carries a big weight. If you average within each sequence first, those few-token samples count as much as everyone else's, and the batch loss gets jumpy. Averaging over all tokens in the batch weights each sample by how many tokens it contributes, which is steadier. 28 00:06:56,869 --> 00:06:58,866 [Hal Turing] What mattered most in their ablation? 29 00:06:58,866 --> 00:07:22,504 [Dr. Ada Shannon] Two things. Start from block attention alone, in diffusion mode, and the average accuracy is 54.2. Add the two-stage schedule, about a trillion AR-only tokens and then 300 billion joint tokens, and it goes to 62.8. Keep the AR loss during the joint stage and it reaches 70.3. So most of the diffusion-mode quality comes from the AR side of the recipe, not from the diffusion machinery. 30 00:07:22,504 --> 00:07:27,426 [Hal Turing] Fine, that's training. At inference, AR mode is just AR. What about diffusion mode? 31 00:07:27,426 --> 00:07:41,173 [Dr. Ada Shannon] It fills a block of masks and commits every position whose confidence clears a threshold. They also train a small sampler to predict whether each top-1 guess will match what the model would have committed anyway, and use it to decide how many positions to commit. 32 00:07:41,173 --> 00:07:42,984 [Hal Turing] And self-speculation is the third mode. 33 00:07:42,984 --> 00:08:14,749 [Dr. Ada Shannon] Linear self-speculation. Append k mask tokens to the verified text and let the diffusion pathway draft them in one pass. Then a second pass runs the same weights causally over the draft. You keep the longest prefix where the two agree, plus one corrected token. There's also a quadratic variant that drafts and verifies in one pass using a bigger attention mask. It reaches higher TPF, but the paper says its kernels are less optimized, so its wall-clock results are weaker. 34 00:08:14,749 --> 00:08:18,092 [Hal Turing] What do they do to make the drafts agree with the verifier more often? 35 00:08:18,092 --> 00:08:44,145 [Dr. Ada Shannon] A LoRA adapter on the draft pathway, only on the attention output projection, about 0.4 percent of the parameters. The loss mixes distribution matching with a total-variation term. That term equals one minus the probability a drafted token gets accepted, so the drafter is directly trained to be accepted. It also trains only up to the first rejected position, because later positions depend on a continuation that never happens at inference. 36 00:08:44,145 --> 00:08:46,746 [Hal Turing] Is that acceptance rule the sampling one, or greedy? 37 00:08:46,746 --> 00:08:56,173 [Dr. Ada Shannon] The loss borrows the sampling-style acceptance probability. But the verification they evaluate is greedy match: the draft token has to equal the AR argmax. 38 00:08:56,173 --> 00:08:59,284 [Hal Turing] Now the speed-of-light analysis. What is it a ceiling of? 39 00:08:59,284 --> 00:09:14,424 [Dr. Ada Shannon] It's an oracle for diffusion mode alone, with no verifier. First, decode a block serially, one position per pass, to define the target sequence. Then search for the largest set of positions you can commit per pass and still land on that same target. 40 00:09:14,424 --> 00:09:17,257 [Hal Turing] And where does linear self-speculation land against it? 41 00:09:17,257 --> 00:09:39,734 [Dr. Ada Shannon] By acceptance rate, close. 6.82 tokens per cycle against 7.60 for the ceiling. By real tokens per forward, far apart: 3.41 against 6.02. Each cycle costs two forward passes, so you divide by two. And it only keeps a prefix, so confident tokens after the first mismatch are thrown away. 42 00:09:39,734 --> 00:09:42,427 [Hal Turing] So that's where the six-times number gets complicated. 43 00:09:42,427 --> 00:10:18,047 [Dr. Ada Shannon] The paper reports 5.99 tokens per forward for linear self-speculation across ten tasks, versus 1.00 for Qwen3-8B. But it doesn't say in that table whether the 5.99 already accounts for the two passes. It's also a different dataset from the 3.41, so I wouldn't force them to reconcile. For wall-clock, at batch size 1 on a GB200 with SGLang, AR runs at 256 tokens a second and self-speculation at 851. About three and a third times faster. With a CUDA-optimized kernel it's 1,015, nearly four times. 44 00:10:18,047 --> 00:10:19,254 [Hal Turing] And against Eagle3? 45 00:10:19,254 --> 00:10:37,133 [Dr. Ada Shannon] About 2.4 times faster on the GB200, 2.3 on an RTX Pro 6000, 1.8 on a DGX Spark. Acceptance length tells the same story: about 6.8 tokens with the LoRA, against 2.75 for Eagle3, at draft length 31. 46 00:10:37,133 --> 00:10:40,988 [Hal Turing] What about AR mode? Does the checkpoint still work as a normal model? 47 00:10:40,988 --> 00:11:00,307 [Dr. Ada Shannon] In AR mode it averages 63.6 across ten tasks, against 62.8 for Qwen3-8B. That's under a point, from models with different data and training histories, so call it comparable rather than a win. The same recipe also carries to 3B and 14B and to a vision-language version. 48 00:11:00,307 --> 00:11:07,552 [Hal Turing] Let me put the headline to the test. Does the 6x tokens-per-forward figure measure the same thing as the 4x throughput figure? 49 00:11:07,552 --> 00:11:20,276 [Dr. Ada Shannon] No. 6x counts tokens committed per forward pass, and a self-speculation cycle spends two passes. Throughput is what a user feels, and it also depends on kernels, hardware, and concurrency. 50 00:11:20,276 --> 00:11:22,134 [Hal Turing] So why lead with the TPF number? 51 00:11:22,134 --> 00:11:45,911 [Dr. Ada Shannon] It's the bigger number, and it doesn't depend on hardware, so it travels well in an abstract. The number to compare is batch-size-1 tokens per second. There's also a loose end. The abstract's 4x on SPEED-Bench doesn't match the 3.3x and 3.97x at one concurrent request in the load-sweep figure. The source doesn't explain the gap, so I'd treat it as unreconciled. 52 00:11:45,911 --> 00:11:53,852 [Hal Turing] Then the accuracy side. Part of the pitch is that the tri-mode model matches Qwen3-8B. What does that comparison actually tell us? 53 00:11:53,852 --> 00:12:13,868 [Dr. Ada Shannon] Not much about the tri-mode idea. They start from Ministral3-8B and add roughly a trillion tokens of AR continued pretraining, then 300 billion joint tokens, then SFT. Qwen3-8B, Eagle3, and SDAR each have their own data and recipes. A sub-one-point edge can't be assigned to the objective. 54 00:12:13,868 --> 00:12:15,725 [Hal Turing] Is there any controlled comparison? 55 00:12:15,725 --> 00:12:39,920 [Dr. Ada Shannon] One. Table 3 trains the same model on the same tokens with and without the diffusion loss. The average moves by 0.14 points at base and 0.43 at instruct. Individual tasks swing by roughly two to seven points in both directions. AIME25 gains about 6.7, but AIME24, IFEval, and HumanEval all drop by two to three points. 56 00:12:39,920 --> 00:12:41,871 [Hal Turing] And AIME is only thirty problems per year. 57 00:12:41,871 --> 00:12:54,549 [Dr. Ada Shannon] Right, and it's one run per setting with no seeds or intervals. So the data says the diffusion loss didn't clearly hurt AR accuracy. It does not show the objectives are complementary, which is the stronger claim in the paper's wording. 58 00:12:54,549 --> 00:12:59,518 [Hal Turing] The ablations look more persuasive, though. Adding the AR loss helps diffusion mode a lot. 59 00:12:59,518 --> 00:13:17,723 [Dr. Ada Shannon] They do, and that part is solid. The AR loss and the two-stage schedule account for most of the diffusion-mode gain. But those variants are all scored in diffusion mode, so what's shown is that diffusion mode needs an AR anchor. The AR side only shows that it tolerates a small diffusion loss. 60 00:13:17,723 --> 00:13:20,277 [Hal Turing] Now the Eagle3 comparison. Is that baseline fair? 61 00:13:20,277 --> 00:13:35,973 [Dr. Ada Shannon] It's real, but there are open questions. The paper says Eagle3 was grid-searched. It doesn't say whether the drafting was tree or chain, or what data the drafter trained on. Whether 2.4x survives a stronger-tuned Eagle3 or a different speculator isn't tested. 62 00:13:35,973 --> 00:13:38,017 [Hal Turing] And the verification is greedy match only? 63 00:13:38,017 --> 00:13:48,837 [Dr. Ada Shannon] That's what's evaluated. The paper never shows sampled decoding with the Leviathan rule, so we don't know whether the output distribution stays exact in this setup, or how the speedup behaves under sampling. 64 00:13:48,837 --> 00:13:53,621 [Hal Turing] What about the speed-of-light number? The 76.5 percent gap sounds large. 65 00:13:53,621 --> 00:14:14,286 [Dr. Ada Shannon] It's an oracle. It knows what the serial denoising will produce, and no deployable sampler has that knowledge. Also, the 76.5 percent compares oracle TPF against the halved self-speculation TPF. On the acceptance-rate metric alone, self-speculation reaches about 90 percent of the oracle. The headroom is real but narrower than the headline. 66 00:14:14,286 --> 00:14:15,726 [Hal Turing] So who should care today? 67 00:14:15,726 --> 00:14:41,407 [Dr. Ada Shannon] Someone serving at low concurrency, on-device or a personal assistant, where one checkpoint gives a measured batch-size-1 gain without a separate drafter. The speedup shrinks as concurrency rises, and the paper itself says AR mode is better for high-concurrency cloud serving. And since the controlled data doesn't show the diffusion loss helping AR accuracy, plain AR continued pretraining remains a live alternative. 68 00:14:41,407 --> 00:14:43,311 [Hal Turing] What does the paper leave for future work? 69 00:14:43,311 --> 00:14:57,986 [Dr. Ada Shannon] Closing the gap between the learned sampler and the oracle, aligning the drafter with the verifier, verifying beyond the accepted prefix, and segment-level parallelism. I'd add sampled decoding to that list, since it's the case the paper doesn't test. 70 00:14:57,986 --> 00:15:22,646 [Hal Turing] I'll try the takeaway. One checkpoint can serve as an AR model and as its own self-speculative decoder, with a real batch-size-1 wall-clock gain over an Eagle3 baseline. The boundary is that 6x overstates the speed, and the benefit of the tri-mode idea over simply continuing AR pretraining is small and unproven in the controlled data. 71 00:15:22,646 --> 00:15:25,014 [Hal Turing] Thanks for listening, everyone. See you next time.