DepthBench: How Residual Connections Unlock Computational Depth in Transformers

Keyu Wang, Yangyi Huang, Jiale Kang, David González-Martínez, Weiyang Liu, Shiwei Liu · ELLIS Institute Tübingen, MPI-IS, Tübingen AI Center, CUHK · 2026
arXiv 2609.32534 Interactive page

The Pre-LN stream keeps growing

Every layer adds its update with weight one, so late updates look small next to the stream.

xl+1 = xl + fl(LN(xl))  |  LayerNorm Scaling: branch input × 1/√l
Illustrative
Schematic assumes unit-size, independent updates, so stream scale ∝ √l and update share ∝ 1/√l. Not measured data.

Same parameters, two shapes

Non-embedding params ≈ 12 · layers · d_model² (+ embeddings). Aspect ratio = width ÷ layers.

Reported endpoints, 400M suite

Ten architectures, three families

Click a node. One LLaMA-style backbone for all.

Select an architecture.

How the layers talk to each other

Schematic of wiring, not weights

Does depth help? Depends on the residual design

Scale check (qualitative, from the discussion)

Qualitative summary, no numbers shown

Do late layers get used differently?

32-layer models, last three quarters of layers only. Thresholds are the authors' picks.

Block AttnRes: the dead initial segment

mHC: composed doubly stochastic maps collapse

Reported effective rank of composed residual maps: mHC 1.44–1.65, HC 2.53–2.81.

Illustrative · random 4×4 matrices

Two readings of one problem

Hover a rival explanation to see what evidence would separate them.

Hover or tap a box.

Evidence audit

Qualitative audit from the discussion

References

  1. DepthBench: Measuring How Residual Connections Enable More Computational Depth — Wang et al., 2026
  2. Scaling Laws for Neural Language Models — Kaplan et al., 2020
  3. Scale Efficiently — Tay et al., 2021
  4. MobileLLM — Liu et al., 2024
  5. The Curse of Depth in Large Language Models — Sun et al., 2025
  6. Hyper-Connections / mHC — ByteDance Seed; DeepSeek-AI
  7. Do Language Models Use Their Depth Efficiently? — Csordás, Manning, Potts, 2025
  8. The Unreasonable Ineffectiveness of the Deeper Layers — Gromov et al., 2024
  9. ShortGPT — Men et al., 2024
  10. Tensor Programs V (muP) — Yang et al., 2022
  11. Don't be lazy: CompleteP — Dey et al., 2025
  12. Gemstones — McLeish et al., 2025
  13. Impact of Depth on Compositional Generalization — Petty et al., 2024
  14. Latent Reasoning: A Recurrent Depth Approach — Geiping et al., 2025
  15. Transformers, Parallel Computation, and Logarithmic Depth — Sanford, Hsu, Telgarsky, 2024
  16. Expressive Power of Transformers with CoT / Parallelism Tradeoff — Merrill, Sabharwal
  17. Resolving Discrepancies in Compute-Optimal Scaling — Porian et al., 2024
  18. DenseFormer — Pagliardini et al., 2024
  19. Peri-LN — Kim et al., 2025