1
00:00:01,000 --> 00:00:07,501
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:07,501 --> 00:00:44,839
[Hal Turing] Today's paper is LayerRoPE: Dynamic Depth-wise Magnitude and Angular Superposition, by Shikhar Srivastava and Christopher Kanan at the University of Rochester. It's a 2026 preprint under review. The setup is a reversal. As data moves through a Transformer, hidden-state norms grow by orders of magnitude with depth, and most of the literature calls that a pathology to suppress. The authors ask whether the growth is a defect, or whether it carries a depth-positional code in the normalization gains that can be made explicit, the way RoPE does for token position.

3
00:00:44,839 --> 00:00:58,539
[Dr. Ada Shannon] The claim is that the growth is partly how each layer learns where it sits in the stack. I'd treat that as a hypothesis, and I think the evidence for it is weaker than the headline. But the practical results are interesting, so let's build the background first.

4
00:00:58,539 --> 00:01:01,046
[Hal Turing] Start with the growth itself. Where does it come from?

5
00:01:01,046 --> 00:01:38,848
[Dr. Ada Shannon] The residual stream. Each block computes its output and adds it to a running vector. In a Pre-Norm model that sum keeps accumulating, so the norm climbs. Prior work treats this as a defect. Sun and colleagues found massive activations in 2024: a few channels at a few tokens thousands of times larger than the median. Darcet and colleagues, in Vision Transformers Need Registers, found high-norm artifact tokens in ViTs. And Sun and colleagues' Curse of Depth paper, 2025, argues the variance growth makes deep layers act close to identity, so they contribute little and can be pruned.

6
00:01:38,848 --> 00:01:41,403
[Hal Turing] So the standard view is that the growth is a bug.

7
00:01:41,403 --> 00:01:49,994
[Dr. Ada Shannon] Mostly, yes. Though massive activations are functional. Zero them and the model breaks. So pure pathology was never quite right.

8
00:01:49,994 --> 00:01:52,687
[Hal Turing] Now the normalization piece. What is RMSNorm doing?

9
00:01:52,687 --> 00:02:21,759
[Dr. Ada Shannon] It rescales a token's vector to unit root-mean-square, then multiplies each channel by a learned gain, gamma. Every layer has its own gamma. It's the one learned per-layer scale applied directly to what a block reads from the stream. Here's the point that matters later. Because RMSNorm resets the input scale, a block sees a unit-scale input no matter how large the stream is. So if the stream is big, the block can only keep its relative contribution by writing bigger outputs.

10
00:02:21,759 --> 00:02:25,753
[Hal Turing] Which would make gamma grow with depth, with no positional signal involved.

11
00:02:25,753 --> 00:02:29,329
[Dr. Ada Shannon] Right. That's a rival explanation, and we'll come back to it.

12
00:02:29,329 --> 00:02:32,022
[Hal Turing] And where does the norm sit? I hear Pre, Post, and Peri.

13
00:02:32,022 --> 00:02:51,620
[Dr. Ada Shannon] Pre-Norm normalizes a block's input. Post-Norm normalizes after the add, which bounds the stream but hurts gradient flow in deep stacks. Peri-Norm normalizes both the block's input and its output before it re-enters the stream. Note that Peri-Norm already has a learned per-layer output gain.

14
00:02:51,620 --> 00:02:53,524
[Hal Turing] What are the remedies people have tried?

15
00:02:53,524 --> 00:03:30,722
[Dr. Ada Shannon] Two families. The first damps or bounds the growth. Layer-Norm Scaling multiplies each block's input by a fixed one over root of layer-plus-one. Peri-Norm belongs here, and so do DeepNet and Depth-muP, which rescale residuals and initialization by depth. Also in this family are learned branch gates. ReZero, by Bachlechner and colleagues in 2021, puts a learned scalar on each residual branch, initialized at zero. LayerScale, from Touvron and colleagues' Going Deeper with Image Transformers in 2021, uses a per-channel learned scale, initialized small.

16
00:03:30,722 --> 00:03:31,930
[Hal Turing] And the second family?

17
00:03:31,930 --> 00:03:48,741
[Dr. Ada Shannon] Restructuring the stream. Attention Residuals, Hyper-Connections, and DeepSeek's manifold-constrained version, mHC. Those run several parallel streams or learn how to aggregate earlier layers. LayerRoPE keeps one stream and doesn't restructure anything.

18
00:03:48,741 --> 00:03:50,134
[Hal Turing] So where does RoPE come in?

19
00:03:50,134 --> 00:04:22,410
[Dr. Ada Shannon] RoPE, from Su and colleagues' RoFormer, rotates query and key feature pairs by an angle proportional to token position, across a geometric spectrum of frequencies. The authors do the same along depth, with layer index in place of token position. There's also a structural point. In a standard untied Transformer, every layer has its own attention and MLP weights, so a block could in principle infer its own index implicitly. RoPE's logic is strongest when weights are shared, as in a looped model like Parcae, where one block runs repeatedly.

20
00:04:22,410 --> 00:04:25,707
[Hal Turing] Last piece of background. The results are fit with scaling laws.

21
00:04:25,707 --> 00:04:49,809
[Dr. Ada Shannon] Loss falls as a power law in compute: an irreducible floor plus a term that shrinks as compute to a negative exponent, alpha. Chinchilla gave the compute-optimal token budget. Here they train 58 million to 1.3 billion parameters at four times that budget, about 80 tokens per parameter, and fit a curve per method. A larger alpha means a steeper climb down.

22
00:04:49,809 --> 00:04:51,667
[Hal Turing] Now the observation behind the claim.

23
00:04:51,667 --> 00:05:13,958
[Dr. Ada Shannon] They study 16 open LLMs from 9 families. In the MLP-block norm weights, about 99 percent of dimensions follow one shared trajectory with depth, and about 1 percent are outlier dimensions that supply the rotation. Projected with multidimensional scaling, the per-layer vectors trace an arc, about 143 degrees in LLaMA-2 7B. The authors read that as a depth code.

24
00:05:13,958 --> 00:05:15,212
[Hal Turing] But you're skeptical.

25
00:05:15,212 --> 00:05:41,450
[Dr. Ada Shannon] First, the exceptions. Attention-block norms show no clear depth code, and Qwen2.5, Qwen3.5, and Gemma 7B show weak or none in the MLP block. Second, that arc is a similarity ordering. If neighbouring layers drift smoothly, you get an arc, with no positional intent needed. Whether that drift is a code or just compensation for growth is what the experiments have to separate.

26
00:05:41,450 --> 00:05:45,584
[Hal Turing] Let's get to the construction. What does LayerRoPE actually replace, and with what?

27
00:05:45,584 --> 00:06:24,825
[Dr. Ada Shannon] It throws out the stack of per-layer gain vectors. At each of four sites there is one shared γ: the input and output of the attention block, and the input and output of the MLP block. Each layer's gain is that shared vector times a depth factor. The factor has two parts. The magnitude is e to the r of ℓ, where r is a constant plus a slope times log of layer-plus-one. The rotation takes a base angle that also follows a log-linear schedule in depth. That angle is spread across feature pairs by a RoPE-style spectrum of frequencies. So four scalars per site, instead of a vector per layer.

28
00:06:24,825 --> 00:06:26,033
[Hal Turing] That sounds close to free.

29
00:06:26,033 --> 00:06:51,528
[Dr. Ada Shannon] Parameters go down, by about 90 thousand at one billion against Pre-Norm. The FLOP overhead is at most about 0.013 percent. And that overhead exists for a reason worth saying plainly. On Pre-Norm there is no output gain, so the method adds a multiplier on every sublayer output. That is a per-layer residual-branch gate. On Peri-Norm, which already has a free output gain, it replaces that gain with the structured one.

30
00:06:51,528 --> 00:06:54,222
[Hal Turing] So on Pre-Norm it adds a feature the baseline never had.

31
00:06:54,222 --> 00:07:19,253
[Dr. Ada Shannon] Yes, and that matters for how we read the results. The mechanism has a read side and a write side. Read is the multiplier on what a block sees after normalization. Write is the multiplier on what it adds back to the stream. The learned schedule shrinks reads and grows writes. The write multiplier grows between six and twenty-five times from first layer to last. It starts with a slope of minus 0.5 and turns positive during training.

32
00:07:19,253 --> 00:07:21,343
[Hal Turing] What does that do to the stream itself?

33
00:07:21,343 --> 00:07:46,420
[Dr. Ada Shannon] It widens it. At 1.3 billion parameters, the residual variance ends up about four times Pre-Norm's, fourteen times Layer-Norm Scaling's, and ninety-four times Peri-Norm's. That shows the network chose to let the stream grow when given the freedom. It doesn't show that the growth is what helps. The authors read it as 'regulate the blocks, don't shrink the stream', but that is an interpretation of variance plots.

34
00:07:46,420 --> 00:07:51,343
[Hal Turing] There's a hint in the Layer-Norm Scaling result, right? Its fixed down-scaling gets partly undone.

35
00:07:51,343 --> 00:08:03,046
[Dr. Ada Shannon] The learned γ largely cancels the fixed one over square root of layer-plus-one. The ratio never drops below about 0.75 at the attention input. That is real evidence about what the networks prefer.

36
00:08:03,046 --> 00:08:05,136
[Hal Turing] Now the main experiment. How was it set up?

37
00:08:05,136 --> 00:08:39,501
[Dr. Ada Shannon] LLaMA-style models on C4, from 58 million to 1.3 billion parameters, each at four times the Chinchilla budget. Every method got its own learning-rate sweep, so nobody is stuck with a rate tuned for someone else. They swept up to 554 million, then extrapolated to 1.3 billion with a power-law fit. The steepness of the compute curve ranks the methods. Pre-Norm has an exponent of 0.117, Peri-Norm 0.129, and LayerRoPE 0.139, with Layer-Norm Scaling in between. Post-Norm diverges.

38
00:08:39,501 --> 00:08:42,102
[Hal Turing] And the headline is 3.4 times less compute.

39
00:08:42,102 --> 00:09:13,867
[Dr. Ada Shannon] At 1.3 billion, matching Pre-Norm's loss costs about 30 cents on the dollar of compute. But that ratio is an interpolation on the fitted curve, and the comparison is against Pre-Norm only. The stronger baseline is Peri-Norm. On average zero-shot accuracy at 1.3 billion, LayerRoPE gets 52.6 and Peri-Norm 51.6. That is about one point. Pre-Norm gets 48.0. LAMBADA perplexity, where lower is better, falls from 21.0 for Pre-Norm to 14.0, a third lower.

40
00:09:13,867 --> 00:09:15,120
[Hal Turing] Does it win everything?

41
00:09:15,120 --> 00:09:46,467
[Dr. Ada Shannon] Six of eight tasks at 1.3 billion. It loses BoolQ, 58.6 to Peri-Norm's 61.3. Seeds are the other limit: three seeds, only at the two smallest scales. The 1.3 billion baselines also use an extrapolated learning rate. And the token ratio is fixed, so this isn't a compute-optimal frontier. There's also a 6.7 billion run with a shared, untuned rate on far fewer tokens, where LayerRoPE still has the lowest loss. That's supportive, but it's one run.

42
00:09:46,467 --> 00:09:50,368
[Hal Turing] What about the depth claim? That's where the curse of depth should show up.

43
00:09:50,368 --> 00:10:25,895
[Dr. Ada Shannon] They fix a narrow model, width 128, and push layers from 48 up to 512. LayerRoPE is the only method whose loss keeps improving through 512. Pre-Norm stops improving after 256 and gets worse at 512. Post-Norm diverges from 192 layers, and Peri-Norm and Layer-Norm Scaling degrade past 96. DeepNet improves at every depth but stays above LayerRoPE. But width 128 and about one Chinchilla budget is a stress test, not a realistic model. And 512 layers is a single seed.

44
00:10:25,895 --> 00:10:27,381
[Hal Turing] And the learning-rate result?

45
00:10:27,381 --> 00:10:52,133
[Dr. Ada Shannon] Learning-rate sensitivity is how fast loss degrades as you move off the optimum. Lower is better. On Pre-Norm at 1.3 billion, it drops from 1.16 to 0.11, about tenfold. At 368 million, LayerRoPE's loss is nearly flat across a tenfold range of rates, 4e-3 to 5e-2, where Pre-Norm diverges. The tuned optimum is also four to eight times Pre-Norm's.

46
00:10:52,133 --> 00:10:54,873
[Hal Turing] So is it more stable, or just run at a higher rate?

47
00:10:54,873 --> 00:11:14,285
[Dr. Ada Shannon] Those are entangled, and that is the right question. The sweeps use one seed and the grid caps the largest stable rate. The paper can't cleanly separate a wider basin from a higher optimum. Still, a method that tolerates a ten-times range of rates is lower tuning risk when you scale up. That part holds regardless of why it works.

48
00:11:14,285 --> 00:11:21,948
[Hal Turing] Let's go back to the central claim, that γ is a depth code. Does the paper rule out the rival explanation you raised early on, growth alone?

49
00:11:21,948 --> 00:11:51,066
[Dr. Ada Shannon] No. Under RMSNorm, growth alone would give γ rising monotonically with depth, with no positional signal. The paper's evidence is the shared trajectory across about 99 percent of dimensions, plus the MDS arc of about 143 degrees in LLaMA-2 7B. Neither separates the two readings. MDS just orders vectors by similarity, so any smoothly drifting sequence would produce an arc. And the rotation comes from roughly 1 percent of outlier dimensions.

50
00:11:51,066 --> 00:11:52,784
[Hal Turing] What experiment would separate them?

51
00:11:52,784 --> 00:12:16,886
[Dr. Ada Shannon] Three come to mind. Permute or reinitialize γ across layers in a pretrained model and measure the damage. Decode layer index from γ after regressing out its scale. Or hold stream norm fixed while varying depth. The paper runs none of them. Add the exceptions I mentioned, and what's established is that a smooth depth-dependent schedule helps. That is narrower than saying a depth code exists.

52
00:12:16,886 --> 00:12:24,456
[Hal Turing] There's also the question of why a block would need to be told its depth. In an untied Transformer, every layer has its own weights.

53
00:12:24,456 --> 00:12:53,481
[Dr. Ada Shannon] Right. RoPE solves a real problem, because attention weights are shared across positions and must be told where each token sits. Untied layers already know who they are. The one shared-weight test is the looped model, Parcae, the scaling-laws paper for stable looped language models by Prairie and colleagues, 2026. But LayerRoPE indexes by network position, not loop iteration. So every pass through the loop gets the same γ, and the model is never told which iteration it's on.

54
00:12:53,481 --> 00:12:55,339
[Hal Turing] So what's doing the work in the ladder?

55
00:12:55,339 --> 00:13:23,946
[Dr. Ada Shannon] The ablation points at a learned scalar gate. At 134M on Pre-Norm, giving every layer its own free γ costs 33.8 thousand extra parameters and changes loss by 0.0001. That says the per-layer vectors were redundant. It does not say the layer index mattered. Magnitude alone recovers most of the gain, and rotation alone recovers much less. And this is one scale, with no seeds reported.

56
00:13:23,946 --> 00:13:25,989
[Hal Turing] Which brings up the obvious prior work.

57
00:13:25,989 --> 00:14:04,070
[Dr. Ada Shannon] ReZero is the closest prior for the write-side gate, and LayerScale is the free per-channel version. The paper cites both, but neither is a baseline in the ladder. The missing control is Pre-Norm plus a plain learned scalar output gate, since on Pre-Norm the write side is a gate the baseline never had. Peri-Norm already has a free output gain, so LayerRoPE's smaller edge there is mild evidence that structure beats free parameters. The paper also cites Wang and colleagues, 2026, Negligible in Size, Significant in Effect, on scale vectors in LLMs, but it treats them as free per-layer parameters.

58
00:14:04,070 --> 00:14:06,392
[Hal Turing] What about the looped and ViT results?

59
00:14:06,392 --> 00:14:39,968
[Dr. Ada Shannon] Consistent but small. At 770M, looped validation perplexity goes from 12.49 to 12.09, and the gain shrinks as you add loops. The ViTs gain 0.9 and 0.6 points of top-1, best-over-training, with no variance reported. Recurrent-state norm reaches around 10^9 at 770M, and the paper doesn't examine what that does to numerical precision. The comparisons that would test an iteration code are per-loop residual scaling and the timestep encodings of Xu and Sato, 2024.

60
00:14:39,968 --> 00:14:41,314
[Hal Turing] Anything else you'd flag?

61
00:14:41,314 --> 00:15:00,215
[Dr. Ada Shannon] The paper's own limitations section says it's unclear to what degree the smoothness is beneficial. The ladder stops at 1.3B, plus the one 6.7B run on a reduced budget. Loss results are on C4 only. And nobody tested low-precision training or quantization, where outliers are the known enemy.

62
00:15:00,215 --> 00:15:01,980
[Hal Turing] For a practitioner, what follows?

63
00:15:01,980 --> 00:15:26,965
[Dr. Ada Shannon] It's a cheap drop-in: fewer parameters, negligible FLOPs, and a wider learning-rate basin. That's worth a trial in a small pretraining run. And it's a reason to doubt that stream growth must be suppressed. Before relying on the depth-code story, I'd want ReZero, LayerScale and a scalar-gate-only baseline. Then shuffle tests and decoding probes on pretrained models, an iteration-indexed looped variant, and a quantization study.

64
00:15:26,965 --> 00:15:46,237
[Hal Turing] So the takeaway is this. A structured, depth-dependent read and write schedule on the norm gains beats the baselines it tried, tolerates higher learning rates, and holds up in very deep stacks. What it hasn't shown is that the network uses a depth code, rather than benefiting from learned scalar gating.

65
00:15:46,237 --> 00:15:47,723
[Dr. Ada Shannon] That's the boundary, yes.

66
00:15:47,723 --> 00:15:49,117
[Hal Turing] Thanks for listening, everyone.