AI POST TRANSFORMERS · EPISODE COMPANION

The Unreasonable Ineffectiveness of Deep Transformer Layers

Up to half the layers of a 70B-parameter model can be deleted with almost no drop on standard QA benchmarks. This page visualizes the residual-stream architecture that makes that possible, how the pruning cut point is found, what the healing process costs, and where the paper's own claims get contested.

arXiv:2403.17887 ↗ ICLR 2025 Gromov, Tirumala, Shapourian, Glorioso, Roberts Meta FAIR · UMD · Cisco · Zyphra · MIT · Sequoia
The residual stream is an additive highway

Every layer reads the current state off the stream, computes attention + feed-forward, and adds its contribution back on — it never overwrites. Bar height below each block is that layer's mock contribution magnitude. Deep-middle layers add almost nothing; the final layer is the one hard exception.

High contribution (early layers)
Diminishing contribution (deep-middle)
Critical — never pruned (final block)
Full model: all 16 representative blocks active. Every contribution is summed onto the stream.
Angular distance heatmap — equation 7

For every starting layer l and block size n, the paper measures how similar the input to layer l is to the input at layer l+n, using only the final token of the sequence, averaged over 10,000 C4 samples. Hover a cell for its value. The coolest (bluest) cell is l* — the safest place to cut.

High distance — must keep
Moderate
Low distance — safe to prune
Computing l*…
Exception: any block that includes the very last layer before the LM head spikes back to high distance in every model family tested — that layer is load-bearing regardless of depth.
Flat region, then a cliff — or not

Mock curves illustrating the paper's reported pattern across four models. Toggle between metric families: QA benchmarks show a flat region followed by a sharp collapse; reasoning benchmarks degrade immediately; C4 loss climbs smoothly straight through the point where QA accuracy falls off a cliff.

Stitching the seam back together

After a block is removed, the two remaining halves are spliced directly and fine-tuned to smooth over the mismatch — using QLoRA so the whole operation fits on one GPU instead of a training cluster.

What actually holds up

Hosts push back on the paper's own framing throughout the episode. Bar length is a rough confidence read on each claim after that back-and-forth — hover a bar for the reasoning.

References

  1. The Unreasonable Ineffectiveness of the Deeper Layers — Gromov, Tirumala, Shapourian, Glorioso, Roberts (2024)
    arxiv.org/abs/2403.17887
  2. ShortGPT: Layers in Large Language Models are More Redundant Than You Expect — Men, Xu, Zhang, Wang, Lin, Lu, Han, Chen (2024)
    Google Scholar ↗
  3. Locating and Editing Factual Associations in GPT (ROME) — Meng, Bau, Andonian, Belinkov (2022)
    Google Scholar ↗
  4. Transformer Feed-Forward Layers Are Key-Value Memories — Geva, Schuster, Berant, Levy (2021)
    Google Scholar ↗
  5. Compact Language Models via Pruning and Knowledge Distillation (Minitron) — Muralidharan et al., NVIDIA (2024)
    Google Scholar ↗
  6. The Truth is in There: Improving Reasoning with Layer-Selective Rank Reduction (LASER) — Sharma, Ash, Misra (2023)
    Google Scholar ↗
  7. LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding — Elhoushi, Shrivastava, Liskovich, Hosmer et al. (2024)
    Google Scholar ↗
  8. Are Emergent Abilities of Large Language Models a Mirage? — Schaeffer, Miranda, Koyejo (2023)
    Google Scholar ↗
  9. SliceGPT: Compress Large Language Models by Deleting Rows and Columns — Ashkboos, Croci, Gennari do Nascimento, Hoefler, Hensman (2024)
    Google Scholar ↗
  10. Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks — Yang, Yu, Zhu, Hayou (2023)
    Google Scholar ↗