1
00:00:01,000 --> 00:00:07,733
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:07,733 --> 00:00:36,015
[Hal Turing] Today we're looking at "How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference." It's from Chanakya Ekbote and colleagues at FAIR at Meta and MIT, posted to arXiv in October 2026. The question is how you should loop a transformer's layers when your scarce resource is model size, training compute, or decoding speed. Those three pull in different directions.

3
00:00:36,015 --> 00:00:50,272
[Dr. Ada Shannon] The answer changes with the constraint. Looping saves parameters, but it doesn't save training FLOPs, and it can make decoding slower. The paper tries to say which looping design fits which constraint, and whether a parallel variant can fix the decoding problem.

4
00:00:50,272 --> 00:00:52,316
[Hal Turing] Start with the basics. What is looping?

5
00:00:52,316 --> 00:01:20,923
[Dr. Ada Shannon] You reuse the same transformer weights several times on a token. Loop the whole stack four times and you get the compute depth of a model four times deeper, with the stored parameters of one. Universal Transformers, by Dehghani and colleagues at Google in 2018, tied weights across depth like this. Recent work from Saunshi, Geiping and others revived it for reasoning. The paper's main design is Full-Stack Looping: the last block's output feeds back into the first block.

6
00:01:20,923 --> 00:01:22,176
[Hal Turing] And the other designs?

7
00:01:22,176 --> 00:01:34,344
[Dr. Ada Shannon] Two contrasts. Middle-Stack looping runs the first and last third of the blocks once and loops only the middle group. Per-Layer looping repeats each block L times before moving to the next. L is just the number of loops.

8
00:01:34,344 --> 00:01:37,223
[Hal Turing] So where does the cost show up? You said it doesn't save FLOPs.

9
00:01:37,223 --> 00:01:55,985
[Dr. Ada Shannon] Two places. Training compute grows with L, because every loop is a real forward and backward pass. And at decode time, loop two for a token needs loop one's output for that same token. A model with four loops does four sequential passes per token. Latency grows roughly L-fold, even though the model is small.

10
00:01:55,985 --> 00:02:00,257
[Hal Turing] Decoding is usually bandwidth-bound, though. Aren't the compute units mostly idle?

11
00:02:00,257 --> 00:02:27,053
[Dr. Ada Shannon] Yes, and that's the opening. Cross-Loop Parallelism changes the dependency. Loop two for token n uses loop one's state from token n minus one, which already exists. So all loops for the current token could, in principle, run at once. The Parallel Loop Transformer, from Wu and colleagues in 2025, builds on this. It also shares the KV cache across loops and uses gated sliding-window attention, called G-SWA, to cut cache memory.

12
00:02:27,053 --> 00:02:28,632
[Hal Turing] Where does the paper land on that?

13
00:02:28,632 --> 00:02:58,957
[Dr. Ada Shannon] Later. First the rest of the vocabulary. Tokens per parameter, or TPP, is training tokens divided by parameter count. Around 160 is heavily over-trained, and that's where looping is meant to pay off. The main metric is bits per byte on a DCLM Baseline 1.0 validation subset of about 41 million tokens. It's language-modeling loss normalized by bytes, so tokenizers don't matter. Lower is better.

14
00:02:58,957 --> 00:03:00,536
[Hal Turing] And the adapters in the title?

15
00:03:00,536 --> 00:03:21,666
[Dr. Ada Shannon] Exact weight sharing forces one function to serve every loop. Here each loop gets a small low-rank adapter, rank at most eight, added to the shared weights. Relaxed Recursive Transformers by Bae and colleagues at Google, 2025, did this when converting pretrained models. This paper trains the adapters jointly, from scratch.

16
00:03:21,666 --> 00:03:25,892
[Hal Turing] So the three constraints are parameters, training FLOPs, and inference latency.

17
00:03:25,892 --> 00:03:44,097
[Dr. Ada Shannon] Right. Fixed parameters and fixed FLOPs are two separate training regimes, and latency is the third. The setup is models from 70 million to 1 billion parameters, trained from scratch on up to 155 billion tokens, with up to 32 loops. Only the smaller models go to the highest loop counts.

18
00:03:44,097 --> 00:03:49,762
[Hal Turing] Let's start with where to loop. Is there a winner among Full-Stack, Middle-Stack and Per-Layer?

19
00:03:49,762 --> 00:04:08,571
[Dr. Ada Shannon] It depends on what you're short of. With parameters fixed, Full-Stack looping gives the best quality. With training FLOPs fixed, Middle-Stack gives the best return per FLOP. Per-Layer looping and independent attention or MLP looping don't hold up as loops increase; the independent versions degrade sharply.

20
00:04:08,571 --> 00:04:10,010
[Hal Turing] And more loops keeps helping?

21
00:04:10,010 --> 00:04:24,778
[Dr. Ada Shannon] Up to roughly 16 loops. After that the gains flatten, sometimes reverse, and some runs go unstable. The catch is that the 32-loop evidence comes only from the 70M and 100M models. Larger models were never pushed that far.

22
00:04:24,778 --> 00:04:30,862
[Hal Turing] Now the adapters. If every loop reuses identical weights, does a small per-loop adapter fix that?

23
00:04:30,862 --> 00:04:49,716
[Dr. Ada Shannon] It helps, and more as loops increase. At two loops the gain is almost nothing. The variants are Loop LoRA, Loop Matrix and Loop Softmax, and Loop Matrix is best. At 300M the adapters add about 1 to 4 percent more non-embedding parameters. Doubling the depth adds about 51 percent.

24
00:04:49,716 --> 00:04:51,992
[Hal Turing] So that's cheap. Is it also cheap in compute?

25
00:04:51,992 --> 00:05:05,599
[Dr. Ada Shannon] No. They're parameter-efficient, not compute-efficient. At comparable FLOPs, doubling true depth reaches lower loss than the adapters. So exact weight sharing does limit high loop counts, and a little per-loop freedom relieves part of that.

26
00:05:05,599 --> 00:05:08,385
[Hal Turing] Then the parallel side. What goes wrong with the original PLT?

27
00:05:08,385 --> 00:05:31,327
[Dr. Ada Shannon] The shift is the problem. Loop l at token n now reads loop l-1's state from token n-1. That state was produced while predicting what comes next, so it's a guess about token n, not the token itself. Those errors compound across positions. On top of that, the shifted state and the token embedding have different RMS magnitudes, so one input can swamp the other.

28
00:05:31,327 --> 00:05:33,602
[Hal Turing] So the fix is to put the real token back in.

29
00:05:33,602 --> 00:05:51,342
[Dr. Ada Shannon] Yes. Normalized Shifted PLT re-injects the current token's embedding. It applies RMSNorm separately to the embedding and to the previous-token state, then mixes them with a rank-8 linear gate. Each loop keeps its own KV cache, with no KV sharing and no gated sliding-window attention.

30
00:05:51,342 --> 00:05:54,082
[Hal Turing] Which piece does the work, the gate or the embedding?

31
00:05:54,082 --> 00:06:14,005
[Dr. Ada Shannon] The embedding. A fixed 50/50 mix, a scalar gate, the linear gate and a small MLP gate all perform about the same. Remove the embedding and the shifted model gets worse than a vanilla one, and some runs diverge. Raw shifted looping without the fix also gets worse as loops are added.

32
00:06:14,005 --> 00:06:16,698
[Hal Turing] How close does the fixed version get to ordinary looping?

33
00:06:16,698 --> 00:06:32,209
[Dr. Ada Shannon] At 1B with two loops, it beats Full-Stack Looping at every budget they show. At four loops it's still competitive, but the gap to its own unshifted twin is clearer and widens. So it recovers most of the quality at small loop counts, and the recovery weakens as loops grow.

34
00:06:32,209 --> 00:06:35,367
[Hal Turing] The widening gap is the part I'd worry about. How big is it?

35
00:06:35,367 --> 00:06:40,383
[Dr. Ada Shannon] The text gives no numeric gap, so I wouldn't put a number on it. The direction is clear, though.

36
00:06:40,383 --> 00:06:43,541
[Hal Turing] There's also a data efficiency claim in the over-trained regime.

37
00:06:43,541 --> 00:07:02,117
[Dr. Ada Shannon] Right. A two-loop model trained at 80 tokens per parameter reaches a given loss with fewer FLOPs than a vanilla model trained at 160. Four loops reaches lower absolute loss, but each extra FLOP buys less. Two loops is the efficient point, and four is for when quality matters more than compute.

38
00:07:02,117 --> 00:07:09,036
[Hal Turing] Now the claim that motivates the parallel side. Does Normalized Shifted PLT actually decode faster?

39
00:07:09,036 --> 00:07:30,816
[Dr. Ada Shannon] We don't know. The paper reports no latency, no tokens per second, and no memory-bandwidth numbers. The claim that loop stages can run concurrently is a dependency argument: loop l now reads the state from loop l minus 1 at the previous token, so in principle they can overlap. Every result in the paper is validation loss.

40
00:07:30,816 --> 00:07:34,346
[Hal Turing] And the best variant keeps a separate KV cache for every loop.

41
00:07:34,346 --> 00:08:00,306
[Dr. Ada Shannon] Right, and that's where I'd worry. Running loops together saves weight reads, because you load the weights once for several stages. But with long context or a big batch, decoding is limited by reading the KV cache, not the weights. Per-loop caches multiply those reads by L. So the crossover depends on context length, batch size and L. The paper offers no crossover. That's my inference, not a result.

42
00:08:00,306 --> 00:08:03,046
[Hal Turing] Isn't that what the original design was built to avoid?

43
00:08:03,046 --> 00:08:18,417
[Dr. Ada Shannon] Yes. KV sharing and sliding-window attention exist precisely to cut cache memory. Those variants score lower here. The paper says they approach the per-loop cache, but gives no number, so the quality-versus-memory trade is unquantified.

44
00:08:18,417 --> 00:08:22,040
[Hal Turing] Now the quality claim. The headline says it recovers standard looping.

45
00:08:22,040 --> 00:08:46,978
[Dr. Ada Shannon] At small loop counts. At 1B and two loops it beats Full-Stack Looping. At four loops the paper itself says a clearer gap to the unshifted control remains, and that it widens. The 1B runs stop at four loops, the 200M to 500M runs stop at eight. The 32-loop saturation belongs to standard looping on the small models. PLT was never pushed past eight, and long loop chains are exactly where latency would hurt most.

46
00:08:46,978 --> 00:08:51,854
[Hal Turing] So is the win over Full-Stack coming from parallelism, or from the re-injection recipe itself?

47
00:08:51,854 --> 00:09:03,371
[Dr. Ada Shannon] The paper's own comparison answers that: the fair test for parallelism is shifted PLT against the unshifted control that uses the same re-injection and normalization. That is the gap that widens with L.

48
00:09:03,371 --> 00:09:09,594
[Hal Turing] Any word on training cost? Loop l at position n needs loop l minus 1 at position n minus 1.

49
00:09:09,594 --> 00:09:18,186
[Dr. Ada Shannon] The text doesn't address it. I'd want to know whether training is truly parallel across the sequence, and whether the FLOP counts mean the same thing across variants.

50
00:09:18,186 --> 00:09:20,879
[Hal Turing] What about the efficiency claims? How solid are they?

51
00:09:20,879 --> 00:09:38,573
[Dr. Ada Shannon] The two-loop-at-80-versus-vanilla-at-160 comparison sits in a heavily over-trained regime, and the text reports no seeds or confidence intervals. That matters most for the adapters, where the gain at two loops is tiny and can't be separated from noise without a stated noise floor.

52
00:09:38,573 --> 00:09:39,873
[Hal Turing] And the adapters themselves?

53
00:09:39,873 --> 00:09:58,170
[Dr. Ada Shannon] They're tested at one scale, 300M, over three loop counts. A trend over three points is thin. Doubled true depth still wins at matched FLOPs, and I don't see a control with equal extra parameters, like one added unshared layer. So we can't cleanly separate specialization from plain extra capacity.

54
00:09:58,170 --> 00:10:00,214
[Hal Turing] Where does this sit against prior work?

55
00:10:00,214 --> 00:10:27,335
[Dr. Ada Shannon] The Relaxed Recursive Transformers work added layer-wise LoRA when converting a pretrained model. This paper trains the adapters jointly from scratch. That's a real difference, but an incremental one. Mixture-of-Recursions, also from Bae and colleagues, varies depth per token. It attacks looping cost differently and isn't compared here. And Yang and colleagues, in 2026, already reported PLT gains shrinking as loops are added, which fits the four-loop gap.

56
00:10:27,335 --> 00:10:29,471
[Hal Turing] What's missing on the evaluation side?

57
00:10:29,471 --> 00:10:57,103
[Dr. Ada Shannon] Evidence is validation loss, mainly on one DCLM subset. Downstream and other-corpus results are in the appendix and described as consistent, but the text reports no uncertainty or seed variance. The authors' own hypothesis for the divergence without the embedding is error accumulation across positions, yet nobody measures actual generation. And some runs diverged; we aren't told how many. Part of the recovery might be a stability fix.

58
00:10:57,103 --> 00:11:00,493
[Hal Turing] So if someone asks what this paper actually settles, what do you tell them?

59
00:11:00,493 --> 00:11:25,199
[Dr. Ada Shannon] That looping isn't one design choice. You match it to whatever you're short of. If parameters are the limit, loop the full stack. If training FLOPs are the limit, loop a middle block. If you plan many loops, per-loop adapters start to earn their small cost. For parallel looping, re-inject the current-token embedding and normalize it separately from the previous-token state. Those are controlled findings from training from scratch.

60
00:11:25,199 --> 00:11:26,313
[Hal Turing] And the boundary?

61
00:11:26,313 --> 00:11:43,542
[Dr. Ada Shannon] The inference benefit is untested. The motivation for the parallel variant is faster decoding, and nobody measured decoding. The quality recovery is shown only up to four loops at 1B and eight at smaller scales. So you have a quality result and a dependency argument. You don't have a speed result.

62
00:11:43,542 --> 00:11:47,304
[Hal Turing] Then what's the decision rule for someone tempted by the parallel version?

63
00:11:47,304 --> 00:12:12,242
[Dr. Ada Shannon] Don't pick it for latency until you've measured decode on your target hardware, at your real context length and batch size, with the per-loop KV cache counted. That cache is the part that could eat the savings, though that's my inference, not a paper result. As for where looping looks best, the evidence points to over-trained, parameter-limited settings. If you're compute-limited in training, remember every loop is paid for in FLOPs.

64
00:12:12,242 --> 00:12:14,100
[Hal Turing] What do the authors say comes next?

65
00:12:14,100 --> 00:12:35,137
[Dr. Ada Shannon] Two things. One is combining parallel looping with speculative decoding. The other is theory: how normalization and embedding re-injection affect stability, both when the recurrence runs across tokens and within the same token. Right now the fix is empirical. The embedding matters, the gate barely does, and there's no account of why.

66
00:12:35,137 --> 00:12:36,438
[Hal Turing] What would you want added?

67
00:12:36,438 --> 00:13:04,209
[Dr. Ada Shannon] A latency and KV-bandwidth benchmark with a real crossover analysis, sweeping context length, batch size and loop count. Then push the parallel variant past eight loops and to larger scale, to see whether that gap keeps widening. And a comparison against Mixture-of-Recursions style adaptive depth, where the model chooses how many loops each token gets. A fixed loop count is the simplest setting, and it may not be the best one.

68
00:13:04,209 --> 00:13:10,432
[Hal Turing] That seems like the natural rival. Fixed depth for everything versus spending loops only where a token needs them.

69
00:13:10,432 --> 00:13:14,797
[Dr. Ada Shannon] Right, and the two aren't exclusive. But this study can't tell us how they stack up.

70
00:13:14,797 --> 00:13:28,497
[Hal Turing] Let me try the short version. How you loop should follow your binding constraint, and the current token has to stay in the loop for parallel looping to hold its quality. What the paper doesn't show is that the parallel version is actually faster.

71
00:13:28,497 --> 00:13:30,076
[Dr. Ada Shannon] That's it. Finding plus limit.

72
00:13:30,076 --> 00:13:33,977
[Hal Turing] Thanks for listening, everyone. And thanks, Ada. Until next time.

73
00:13:33,977 --> 00:13:35,091
[Dr. Ada Shannon] Thanks, everyone.