1
00:00:01,000 --> 00:00:43,424
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

Today's paper is "Transformer Layers as Painters," by Qi Sun and colleagues: Marc Pickett, Aakash Kumar Nain, and Llion Jones, so Sun et al. with three co-authors. It comes from Emergence AI, Sakana AI, and the Institute of Science Tokyo, and the arXiv version we read is stamped February 2025. Here's the tease. Take a pretrained Llama 2, freeze it, and then skip layers, shuffle them, or run several at once. No retraining, no fine-tuning. It doesn't fall over. It gets worse, gradually, and that's a strange thing for a 32-layer stack to do.

2
00:00:43,424 --> 00:01:26,574
[Dr. Ada Shannon] The finding is that the middle layers are much more uniform than anyone should expect, and that frozen models trade accuracy for depth smoothly instead of breaking. That matters because depth is latency. If you know which layers you can drop or run side by side, you have a dial for conditional computation. The authors started somewhere more modest. They wanted to know whether several layers could be merged into one bigger layer.

The paper frames it with a painter analogy. The canvas is the hidden state. Each layer is a painter on an assembly line who either adds a few strokes or passes the canvas along through the residual connection. All the painters share one vocabulary for what a painting looks like, so a painter can take a canvas from earlier in the line without catastrophe. The authors say outright that this is a thinking tool, not a rigorous theory.

3
00:01:26,574 --> 00:01:44,924
[Hal Turing] Oh wait, wait, wait, hold on. "Share one vocabulary" is carrying the whole paper, isn't it? In a CNN, layer five's channels are in some arbitrary basis that layer six has learned to read. Hand layer six the wrong canvas and you get garbage. Why would a transformer be different?

4
00:01:44,924 --> 00:02:29,399
[Dr. Ada Shannon] Because of the residual stream. Each layer computes x plus an update, so every layer reads and writes the same d-dimensional space, and the coordinate system is pinned. Veit and colleagues showed the image-domain version in 2016 at Cornell, where deleting single blocks from a ResNet hurt only modestly while deleting from a VGG-style network was catastrophic.

For transformers, we have tools for reading that shared space. nostalgebraist's 2020 logit lens applies the final unembedding to intermediate states. Belrose and colleagues, from EleutherAI and Berkeley in 2023, showed that's biased, so their tuned lens trains a small affine translator per layer. Predictions refine smoothly with depth. This paper leans on cosine similarity between layers, and Kornblith's CKA from Google Brain in 2019 is the standard cousin.

5
00:02:29,399 --> 00:02:39,549
[Hal Turing] Here's what I genuinely don't know. If the middle is that uniform, has anyone just cut layers out and measured? Do we already know how many are dead weight?

6
00:02:39,549 --> 00:03:27,849
[Dr. Ada Shannon] Yes, concurrently, and the picture agrees. ShortGPT and Shortened LLaMA, the latter from Nota AI, do depth pruning and then heal with light LoRA fine-tuning. Gromov and colleagues, in "The Unreasonable Ineffectiveness of the Deeper Layers," from Meta and MIT in 2024, pick the block whose input and output are most similar and remove it. Question-answering accuracy holds until a sharp collapse, while perplexity degrades smoothly. That matters for what follows, because the metric changes the story.

The closest study is Lad, Gurnee, and Tegmark at MIT in 2024, "The Remarkable Robustness of LLMs: Stages of Inference?" They delete or swap layers on frozen models and propose four stages: detokenization, feature engineering, prediction ensembling, and residual sharpening. Painters describes the same lower, middle, and final split, but only empirically.

7
00:03:27,849 --> 00:03:35,399
[Hal Turing] And the reordering and parallel side? Presumably someone tried making the model tolerate that on purpose.

8
00:03:35,399 --> 00:04:16,625
[Dr. Ada Shannon] Several people. LayerShuffle, from the IT University of Copenhagen in 2024, trains with random layer order. CQIL, concurrent quasi-independent layers, runs layers side by side. DenseFormer, from EPFL in 2024, adds averaging across depth. Switch Transformers, from Google in 2021, is the routing analogy, since it picks per token which experts run. The looped-parallel test connects to recurrent-depth and looped language models, where you reuse a block for several passes.

The difference is that those methods bake the property in at training time. Painters asks whether an ordinary sequentially trained model already has it, and the answer is that it partly does.

9
00:04:16,625 --> 00:04:20,800
[Hal Turing] So what's actually being run? Which models, which tests?

10
00:04:20,800 --> 00:05:07,600
[Dr. Ada Shannon] The main study is Llama2-7B, 32 layers, with 13B and 70B for scaling, and BERT-Large, 24 layers, an encoder. Mistral-7B and Pythia-6.9B sit in the appendix. For Llama2 the benchmarks are ARC, HellaSwag, WinoGrande, LAMBADA, and GSM8K. Each is normalized so zero is random or max-class guessing and one is the full unmodified model, and the headline is the median across the five. BERT uses GLUE with an unnormalized average, and it gets the standard fine-tuning step, so it can adapt.

The paper asks eight questions: shared space, necessity, redundancy, order, parallelism, task dependence, looping, and least-harmful variant. Every one is empirical. It offers no theory for why the robustness exists, and it says so.

11
00:05:07,600 --> 00:05:17,275
[Hal Turing] Okay, answers. Take a single layer out of Llama2-7B and score LAMBADA. I'm guessing it depends heavily on which layer you pull.

12
00:05:17,275 --> 00:05:55,500
[Dr. Ada Shannon] Entirely. Skip or swap a layer at either edge and LAMBADA collapses. Do it in the middle and you barely see a dent. They even feed layer 4's output to layer 6, then send layer 6's output to layer 5, and the model shrugs. The cosine heatmaps show block structure that lines up with that curve. In 13B the groups are layer 0, layers 1 to 3, a big middle, then the final one or two. High similarity alone is weak evidence, and the authors say so. The heatmap matching the performance figure is what they lean on. Separately, in the Pythia appendix, from 14M to 12B parameters, the beginning and middle classes grow with depth. The ending class stays a single layer at every size.

13
00:05:55,500 --> 00:06:05,875
[Hal Turing] Now I'm genuinely curious about the Pythia result. Why would the ending stay exactly one layer from 14 million to 12 billion parameters?

14
00:06:05,875 --> 00:06:35,200
[Dr. Ada Shannon] The paper doesn't say. It's an observation with no mechanism attached. On Skip, read the curves left to right, from few layers removed to many. Llama2-7B and BERT-Large both degrade gracefully. Plot by percentage of layers skipped and 7B, 13B and 70B trace similar trends. Fine-tuning after Skip helps under about 30 percent skipped and hurts beyond that, though that's ARC only, in Appendix A.2. Here's the catch, though. If the middle is uniform, you'd expect that replacing it with copies of the center layer would be harmless.

15
00:06:35,200 --> 00:06:43,200
[Hal Turing] That was my next question. Same vocabulary, same space, so sixteen copies of layer 16 should be fine.

16
00:06:43,200 --> 00:07:03,850
[Dr. Ada Shannon] It's the worst variant by far. Middle Repeat hits random baseline quickly, much worse than Skip. Appendix A.4 says repetition pushes hidden states out of the shared space. Cosine similarity drifts, and variance blows up in the last layers. So the middle is uniform but not redundant. Skip keeps the geometry intact. Repeating one layer wrecks it.

17
00:07:03,850 --> 00:07:08,625
[Hal Turing] One painter, sixteen wheels, and a canvas that's just tires.

18
00:07:08,625 --> 00:07:49,875
[Dr. Ada Shannon] Right, and the painters above her have never seen that canvas. Order is the next result. Reversed and Random Order both degrade gracefully, and both beat Skip. So layers still contribute when they see input from an unexpected source. Random beats Reversed, and the authors' reason is that reversal is maximally inconsistent with training order. Random is averaged over 10 seeds. Parallel, averaging the middle window's outputs, beats Skip but loses to Reversed, and every benchmark holds up except GSM8K. That's the task split. ARC and GSM8K fall fastest under Skip, Reversed and Parallel, while HellaSwag and WinoGrande are more robust, so the authors read reasoning as more order-dependent than—

19
00:07:49,875 --> 00:07:55,975
[Hal Turing] Sorry, but the Mishka example is where this gets concrete, right? I want to hear that one.

20
00:07:55,975 --> 00:08:23,550
[Dr. Ada Shannon] Appendix A.7. Mishka buys three sets of shorts, pants and shoes. Parallel Llama2-7B, N equals 14, sets up the correct multiplications, then executes them badly. 16.50 times 3 comes out as 50.50 instead of 49.50. The other products are off too, and the final sum is 192.50 where the full model gets 243. So the structure survives and the arithmetic doesn't. That's their evidence that math is order-dependent, and it's a single example.

21
00:08:23,550 --> 00:08:28,775
[Hal Turing] The right equation, and then it loses the calculator. What about looping?

22
00:08:28,775 --> 00:09:02,925
[Dr. Ada Shannon] Looped Parallel 3X significantly improves on one parallel pass, and it's the best variant for both BERT and Llama2-7B. The single-layer point is a special case, since it just equals repeating one layer three times. Figure 10 sweeps iterations from 1 to 28. The best count grows roughly linearly with the number of parallelized layers, except near M equals 29 and 31. Figure 11 ranks everything. Looped Parallel and Random Order do the least damage and Middle Repeat the most. Appendix A.6 adds that internal looping doesn't beat the base model. It stays below.

23
00:09:02,925 --> 00:09:06,050
[Hal Turing] Do the results travel beyond Llama?

24
00:09:06,050 --> 00:09:44,250
[Dr. Ada Shannon] Mistral-7B, from Jiang and colleagues at Mistral AI in 2023, tracks Llama2-7B closely. Pythia-6.9B, from Biderman and colleagues at EleutherAI in 2023, is less robust, especially to order changes. In A.9, with a frozen BERT and only a tuned head, Looped Parallel sometimes exceeds the full baseline. Then there's the latency claim. The Discussion says Parallel at N equals 8 for Llama2-7B should be about half the latency, assuming enough memory to run layers simultaneously. That's the authors' estimate, not a measurement. Per-task curves appear only as plots, with no tables.

25
00:09:44,250 --> 00:10:07,650
[Hal Turing] Ada, I want to poke the headline. The paper admits GSM8K collapses under Skip, Reversed and Parallel. With five tasks, the median is just the third-ranked score. Two benchmarks can sit at zero and the curve still looks healthy, like a jury verdict where two jurors have left the room. So is graceful degradation a property of the model, or of the aggregation?

26
00:10:07,650 --> 00:10:52,125
[Dr. Ada Shannon] Partly the aggregation. The paper's own answer to the parallel question is 'yes, except for math', which is honest, but the abstract leads with graceful degradation. And the N=8 Parallel point behind the roughly two-times latency claim exists only in plotted curves. What GSM8K, ARC, WinoGrande, HellaSwag and LAMBADA each score there is not given in the paper text, so I won't guess. Normalization adds noise too. WinoGrande's random floor is fifty percent and the model sits not far above it, so a small absolute drop becomes a large normalized one. Figure 13 reaches about 1.4 on the y-axis, meaning scores above the full model. The deterministic variants have no confidence intervals, so the ranking of Reversed against Parallel has unknown noise. And the fine-tuning crossover is ARC only, one epoch, one learning rate.

27
00:10:52,125 --> 00:11:06,625
[Hal Turing] Sorry to cut in, but back up to that latency number. Small-batch decode is memory-bandwidth-bound. Every weight still gets read whether layers run in sequence or in parallel, so where does half actually come from?

28
00:11:06,625 --> 00:11:52,350
[Dr. Ada Shannon] On one device it doesn't. This is my analysis, not the paper's. Averaging does identical FLOPs and weight reads, so it saves wall-clock only if hardware runs about sixteen layers at once, and across devices you pay averaging communication per token. Looped Parallel 3X puts sequential passes back and evaluates the middle three times, which is more FLOPs than the original. Skip is the variant that really cuts compute and KV cache, and it's worst on GSM8K and ARC. Also, averaging scales each layer's delta by one over M, so a plain sum or a learned scalar would separate parallelism from that choice. Compare LayerSkip, from Mostafa Elhoushi at Meta in 2024. It trains with layer dropout and gets real speedups. Draft & Verify, by Jun Zhang at Zhejiang University in 2023, uses a skipped frozen model as the draft, with exact outputs.

29
00:11:52,350 --> 00:12:00,350
[Hal Turing] Then the geometry. The three-class story leans on cosine similarity of hidden states. How much would you trust it?

30
00:12:00,350 --> 00:12:45,475
[Dr. Ada Shannon] Not much alone. Massive Activations, by Mingjie Sun at Carnegie Mellon in 2024, shows a few huge dimensions persisting through the middle of the residual stream, and those inflate cosine. The paper cites Godey on anisotropy and then leans on it anyway. Mean-center, drop those dimensions, or use CKA, and see whether the Figure 3 blocks survive. The skip evidence is circular, because robustness is used to support the hypothesis meant to explain robustness. And given the ResNet result from earlier, this may be residual structure, not transformer-specific. Appendix A.4 also sits awkwardly with 'different functions'. If one repeated layer exits the distribution, uniformity may only mean small updates against the residual norm. Measure that ratio. And Wenfang Sun's Curse of Depth, out of Westlake in 2025, blames Pre-LN, which every tested model uses.

31
00:12:45,475 --> 00:12:51,975
[Hal Turing] Here's what I'm curious about. If benchmarks stay flat, what could be silently breaking underneath?

32
00:12:51,975 --> 00:13:33,925
[Dr. Ada Shannon] Factual recall and long generation. Kim and colleagues at Nota AI found pruned models fail at factual QA and long responses even after retraining. Mor Geva at Tel Aviv University put facts in feed-forward layers in 2021, so HellaSwag can stay flat while recall goes. Gromov selected layers by angular distance, so this symmetric window may understate what skipping can do. Csordás at Stanford, 2025, finds later layers add little composition, which fits GSM8K's fragility. Then there's scope. It's mostly Llama2-7B, BERT is fine-tuned, the frozen-head result sits in the appendix, Pythia disagrees on order, and the text contradicts itself on how beginning layers scale.

33
00:13:33,925 --> 00:13:37,100
[Hal Turing] So who should actually do what with this?

34
00:13:37,100 --> 00:13:57,925
[Dr. Ada Shannon] Classification and semantic completion can tolerate skipping. Multi-step arithmetic can't. Layer routing is the natural next step, and Mixture-of-Depths, by David Raposo at Google DeepMind in 2024, is the trained version. The follow-ups are clear. Thaw and fine-tune under the new orders, choose windows by similarity, and test long generation and reasoning.

35
00:13:57,925 --> 00:14:21,075
[Hal Turing] So the takeaway: middle layers are robust but not interchangeable, and tasks differ sharply in fragility. Graceful degradation is partly the median talking, and the latency win is a projection, not a measurement. Settling it takes per-task tables at N=8, wall-clock on real hardware, and long-generation tests. Thanks for listening, and goodbye.