1
00:00:01,000 --> 00:00:42,725
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon. Today we're reading "Full-bandwidth transformer" by Xi Wang and colleagues, eight authors from Johns Hopkins, Princeton, and Microsoft, posted to arXiv on the ninth of August 2026. Xi Wang and Ziyang Cai did the work as Microsoft interns, and John Langford is the senior author. The question: can a transformer feed its full top-layer hidden state, not just the sampled token, back into the bottom of the stack at every decoding step? And can you train that in parallel, so the extra per-token compute acts like more pretraining data?

2
00:00:42,725 --> 00:01:09,924
[Dr. Ada Shannon] Every decoding step, the model builds a vector of thousands of floats at the top of the stack, then keeps about seventeen bits of it: one sampled token. The rest is discarded. The paper calls that the narrow vertical channel. The motivation is scaling laws. Kaplan and colleagues at OpenAI showed in 2020 that loss falls predictably with parameters and training tokens. But high-quality unique tokens are running short. So the question is whether more compute per token can stand in for more tokens.

3
00:01:09,924 --> 00:01:14,150
[Hal Turing] So the model writes a novel and keeps the ISBN.

4
00:01:14,150 --> 00:01:16,825
[Dr. Ada Shannon] Yes. A very expensive ISBN.

5
00:01:16,825 --> 00:01:25,125
[Hal Turing] Okay, but I don't get one thing. The KV cache stores all the past computation. Isn't that information still there?

6
00:01:25,125 --> 00:01:47,525
[Dr. Ada Shannon] It's there, but at the wrong height. Horizontally, attention reads every earlier position, so that axis is full bandwidth. Vertically, a state computed at layer l can only be read by layers above l. A shallow layer at the new token sees shallow, half-processed states from the past, and never what layer thirty concluded. The paper counts reachable states as roughly T times l. And the top layer's output is never cached at all, because nothing above it would read it.

7
00:01:47,525 --> 00:02:00,475
[Hal Turing] Oh wait wait wait— so anything the model didn't write down as a token can't go back to the bottom for more processing? It either narrates its thinking or recomputes it from scratch at every position?

8
00:02:00,475 --> 00:02:17,775
[Dr. Ada Shannon] Exactly. That's why chain of thought works. Zhiyuan Li and colleagues, out of Stanford and Google in 2024, showed that generated tokens are what give a fixed-depth transformer serial depth. The token loop is the only depth-extending channel, and it's seventeen bits wide.

9
00:02:17,775 --> 00:02:26,825
[Hal Turing] Then this is an RNN in a transformer costume. Carry the hidden state forward. Transformers won by dropping exactly that.

10
00:02:26,825 --> 00:02:47,275
[Dr. Ada Shannon] I disagree with that read, Hal. An RNN overwrites a fixed-size state at every step. That makes inference cheap, but training is sequential, and that's why transformers won: teacher forcing trains every position in parallel. Here, nothing gets overwritten. Every earlier state stays in the KV cache, and the fed-back vector is one extra input on top of full attention over the past.

11
00:02:47,275 --> 00:02:54,925
[Hal Turing] But the sequential-training problem comes back anyway. Position t needs the finished pass at t minus one.

12
00:02:54,925 --> 00:03:15,375
[Dr. Ada Shannon] Yes, that part is real, and it's the actual tension. The Feedback Transformer by Angela Fan and colleagues at Facebook AI Research, 2020, gave every layer access to the top of the stack from past steps. It paid for that with sequential training. So we agree on the cost. I'm saying the form is recurrent but the memory isn't, and that changes what the model can lean on.

13
00:03:15,375 --> 00:03:23,801
[Hal Turing] Fair, that's a real distinction. Next piece: the fusion uses a gated linear unit, and I only half remember those.

14
00:03:23,801 --> 00:04:00,026
[Dr. Ada Shannon] Generic version: one linear path, the value, multiplied elementwise by a sigmoid-gated path. Dauphin and colleagues at Facebook AI Research introduced it for language modeling in 2016, and Noam Shazeer at Google put GLU variants into transformer feed-forward blocks in 2020. So your model already has them. The paper's particular use of one is next. There's also a neighbor to place. Coconut, by Shibo Hao and colleagues at Meta in 2024, feeds the hidden state back in place of tokens. This paper augments the token instead. That should make supervision easier, and probably costs more tokens.

15
00:04:00,026 --> 00:04:05,526
[Hal Turing] Give me the headline claims, and we'll treat them as claims. Not as gospel.

16
00:04:05,526 --> 00:04:27,926
[Dr. Ada Shannon] Roughly 1.5 times more tokens' worth of quality in the abstract, matching baselines trained on twice the data and approaching five times on some tasks in the body, negligible per-token decoding cost, and shorter reasoning traces. Latent feedback decoding means the previous top-layer state is fused with the sampled token's embedding and becomes the next input. Later we check whether those numbers survive scrutiny.

17
00:04:27,926 --> 00:04:39,851
[Hal Turing] Right, so here's my real question. If every position now depends on the model's own hidden state at the position before, how do you train that without giving up parallel training?

18
00:04:39,851 --> 00:05:08,201
[Dr. Ada Shannon] In passes instead of positions. Pass one is ordinary. Pass two shifts every top-layer state one position right, fuses it with the token embeddings, and reruns the stack in parallel, since everything it needs finished in pass one. It's Jacobi-style, like PonderLM-2 from Zeng and colleagues in 2025. The loss lands on every pass at weight one, and the gradient is not detached, so later losses backpropagate into earlier states. That's an auxiliary objective, with extra memory, at about k times teacher forcing.

19
00:05:08,201 --> 00:05:22,027
[Hal Turing] But decoding runs this loop for thousands of tokens, and you trained on two or three passes. I'm genuinely curious how that doesn't diverge. And why a gate at all, instead of just adding the state to the embedding?

20
00:05:22,027 --> 00:06:07,852
[Dr. Ada Shannon] Scheduling, and the gate. Most training is single-pass, started from a standard checkpoint; two-pass batches arrive mid-training, then a small share of deeper ones. In Fig. 3, 75% one-pass and 25% two-pass diverges past its trained depth. Add 3% three-pass batches, 75/22/3, and the hidden-state change decays to a plateau, stable out to 1,000 passes: a contraction toward a fixed point. A prefix mixin reverts a random prefix to plain embeddings, matching plain prompt, fused generation. Depth scaling, RMSNorm on the fused input, tied embedding and readout, and jitter noise of 0.02 help too. The gate is W_U times h, elementwise with sigmoid of W_G times e: state on the value path, token only as gate. Additive fusion lets the model zero the state and recover the plain token. Here, dropping the state drops the input.

21
00:06:07,852 --> 00:06:15,652
[Hal Turing] That thousand-pass sweep is a tidy piece of empiricism. So what does serving cost, and what does it not buy?

22
00:06:15,652 --> 00:06:37,402
[Dr. Ada Shannon] The paper's claim is free to serve: under 1% per token, two D-by-D matmuls, KV cache untouched, two lines changed in the decode loop. vLLM uses a persistent hidden-state buffer, like EAGLE and multi-token prediction. There's no mutable register, since earlier states stay in the cache, and no extra decoding depth. The carried state is a deterministic function of the tokens, so—

23
00:06:37,402 --> 00:06:46,102
[Hal Turing] Wait wait, if the tokens already determine it, nothing new comes in. Then why does it help? I'd have guessed information was the point.

24
00:06:46,102 --> 00:07:47,502
[Dr. Ada Shannon] Nothing new comes in. The gain is accessibility. The model could recompute that state, but only by spending depth at every position. Now every layer, the lowest included, reads the past as the full stack processed it: Theta of T times L reachability, not T times l. The paper adds an implicit scratchpad, draft space, which predicts shorter rollouts. Setup: 1B parameters, 24 layers, D of 1,536, 100,352-token tied vocabulary, Phi-4 mix, 8K context, 300K-token batches, 1.2M for the 1T baseline. Token-equivalent compute is training tokens times average passes per batch. Token-matched holds tokens equal; FLOP-matched holds that product equal. The table: 10B tokens, all three-pass, 40B equivalent; 100B at 75% one, 25% three, 150B; 200B and 400B at 75/22/3, 256B and 512B. Roughly 4x, 1.5x, 1.28x, 1.28x.

25
00:07:47,502 --> 00:07:51,252
[Hal Turing] Start with prefill. Where does the gain show up?

26
00:07:51,252 --> 00:08:28,227
[Dr. Ada Shannon] Front-loaded, mostly at the first fused pass. At step 0, with no feedback, there's a small validation-loss cost, but average LM Eval accuracy is already up. With two feedback passes, 100B FB reaches the 200B standard baseline and 200B FB reaches 400B, which the paper calls roughly 2x data efficiency. In Table 2, 200B with one pass scores 53.58 zero-shot, 52.66 with none, against TinyLlama at 2T tokens, 53.23, and Llama 3.2 1B, 55.31.

27
00:08:28,227 --> 00:08:33,927
[Hal Turing] Multiple choice is the easy part. What happens in free-form generation?

28
00:08:33,927 --> 00:09:58,852
[Dr. Ada Shannon] SOFT beats STANDARD on every task, same weights. Math500 at 200B goes 0.27 to 0.37, past the 1T baseline. FUSED leads on code: HumanEval 0.31 to 0.34, MBPP 0.38 to 0.40. After 12B long-context tokens and 6B instruction-tuning tokens, Table 1 reads STANDARD/SOFT/FUSED for FB 200B, FB 400B, then standard 200B/400B/1T. GSM8K: 64.52/67.93/67.55, 67.90/71.00/71.80, then 62.93/68.39/70.13. MATH-500: 43.80/45.60/45.60, 46.00/45.40/48.40, then 42.40/46.40/47.40. HumanEval: 42.54/45.06/45.92, 46.50/47.20/47.60, then 37.16/44.85/50.01. MBPP: 38.39/39.80/41.22, 40.50/40.60/41.70, then 38.61/40.28/41.93. The paper says MBPP FUSED nearly closes the gap to 1T, 41.2 against 41.9.

29
00:09:58,852 --> 00:10:01,702
[Hal Turing] And the shorter reasoning traces?

30
00:10:01,702 --> 00:10:22,777
[Dr. Ada Shannon] On base models, SOFT gives shorter median length at equal or better accuracy on Math500. The worked example is 137 over 500: standard decoding grinds through long division, soft writes 0.274 and answers 4. After instruction tuning it disappears, which the paper blames on off-policy tuning data re-imposing verbosity.

31
00:10:22,777 --> 00:10:37,302
[Hal Turing] And the probes settle it, right? Layer-0 accuracy hits 99.6% on completion tracking and 100% on delayed memory after one recurrent step. So the model is using the state.

32
00:10:37,302 --> 00:10:53,302
[Dr. Ada Shannon] No no, that's not what it shows. Decodable is not used. The paper says so itself: a probe shows the information is present at layer 0, not that the model acts on it. The gain also shrinks with more overwriting, and full recurrence does best.

33
00:10:53,302 --> 00:10:57,652
[Hal Turing] But the downstream numbers went up too. Isn't that the use?

34
00:10:57,652 --> 00:11:05,977
[Dr. Ada Shannon] Those speak to use, and I'd call them the real evidence. The probes only show the door is open. Two claims, two experiments.

35
00:11:05,977 --> 00:11:22,827
[Hal Turing] Ada, let me test the data-efficiency framing in FLOPs. The 100B feedback run costs 150B-equivalent and matches the 200B standard baseline. How much of 'twice the data' is left once you count what training actually cost?

36
00:11:22,827 --> 00:12:04,577
[Dr. Ada Shannon] It shrinks to roughly 1.3 to 1.6x. That's our derivation from the table, not the authors' statement. Two hundred over one-fifty is 1.33. Four hundred over 256 is about 1.56. Three things push it lower still. The two-pass prefill behind the Fig. 4 evaluation is inference cost that token-equivalent compute skips. Gradients aren't detached, so activation memory grows with passes, and the text reports no wall-clock or memory numbers. And the 5x claim, 200B feedback against the 1T baseline, is about 3.9x in FLOPs, on selected tasks. That 1T run also used a 1.2 million token batch against 300K for everything else, so the comparison is—

37
00:12:04,577 --> 00:12:28,452
[Hal Turing] Sorry, hold on, I want to defend something. Math500 going from 0.27 to 0.37 on the same weights, beating the 1T model, is a clean result. Batch size can't explain a gap between STANDARD and SOFT decoding of one checkpoint. I'll say it plainly: that comparison is the tightest design in the paper, and it isn't an accounting artifact.

38
00:12:28,452 --> 00:12:55,427
[Dr. Ada Shannon] I actually disagree with how much it carries, Hal. Look at 400B with feedback unused. GSM8K is 67.90 against 68.39 for the standard baseline, MATH-500 is 46.00 against 46.40, despite 1.28 times the training compute. So the training objective alone buys roughly nothing. And at 400B, SOFT sits below STANDARD on MATH-500, 45.4 versus 46.0. 'Improves on every task' doesn't hold uniformly.

39
00:12:55,427 --> 00:13:03,877
[Hal Turing] But that supports the paper. If the gain lives at decode time, that's exactly the cheap part. Why is that a mark against it?

40
00:13:03,877 --> 00:13:44,452
[Dr. Ada Shannon] Because the evidence is thin where it needs to be thick. One seed per configuration. HumanEval is 164 problems, so two-point gaps sit inside plausible noise, and temperature was grid-searched per regime on the test set. One table row also bothers me. By the paper's own rule a three-pass batch costs 3x, so the 10B run should be 30B token-equivalent, not the listed 40B. Typo or an undisclosed fourth pass, the text doesn't say, so the 4x end isn't clean. And after instruction tuning the 1T baseline still wins HumanEval, 50.01 against 45.9 to 47.6. I'll grant the direction is consistent. The size isn't established.

41
00:13:44,452 --> 00:13:55,752
[Hal Turing] Direction yes, size no, I'll take that. Here's what I genuinely don't know. How does keeping the token compare with Coconut and the loop transformers? Is it the better bet?

42
00:13:55,752 --> 00:14:27,902
[Dr. Ada Shannon] It's a trade. Coconut needs a multi-stage curriculum to supervise its hidden-state replacement. Recurrent depth, by Jonas Geiping out of the ELLIS Institute Tübingen in 2025, reapplies blocks at inference, so cost scales with the number of steps. That's the natural compute-matched foil, and nobody ran it. The Feedback Transformer's finding that top-layer-only memory nearly matches the full mixture supports this design, but it's a 2020 result at another scale. T²MLR and PonderLM-2 are close siblings, and the authors admit they can't say which injection point is best.

43
00:14:27,902 --> 00:14:35,902
[Hal Turing] You called the under-1% serving cost plausible from FLOPs. Does it hold up in practice, and what else worries you?

44
00:14:35,902 --> 00:15:14,202
[Dr. Ada Shannon] The text has no throughput numbers, and decoding is memory-bound. FUSED doubles prefill. My open question is speculative decoding. Verifying drafted tokens in parallel needs each previous hidden state, so does the parallelism survive? And a multi-turn chat that re-prefills the last answer as plain embeddings shows the model a different distribution than it generated under. Then oversight. Tomek Korbak and colleagues at the UK AI Security Institute argued in 2025 that legible reasoning traces are a safety asset. This paper celebrates moving computation out of them, on base models only, and the effect vanishes after tuning.

45
00:15:14,202 --> 00:15:16,727
[Hal Turing] So what would change your mind?

46
00:15:16,727 --> 00:15:37,927
[Dr. Ada Shannon] FLOP-, wall-clock- and memory-matched baselines. A detach-gradient ablation. A multi-epoch baseline in the spirit of Niklas Muennighoff's data-constrained scaling work at Hugging Face in 2023. Larger scale, and on-policy post-training. Those decide whether compute per unique token is a third scaling axis or a one-billion-parameter curiosity.

47
00:15:37,927 --> 00:15:58,477
[Hal Turing] So, the takeaway. Latent feedback looks like a real, cheap-to-serve gain, roughly 1.3 to 1.6x in FLOPs at 1B, with the 2x and 5x numbers reading as best cases. If you serve models, watch the speculative decoding and multi-turn questions. Thanks for listening, and see you next time.