The Sparsity Wall: What Reiner Pope Told Dwarkesh About MoE and Sparse Attention

Episode title: The Sparsity Wall: What Reiner Pope Told Dwarkesh About MoE and Sparse Attention A two-paper deep dive framed around the Dwarkesh Patel x Reiner Pope blackboard lecture on training and serving frontier LLMs. The hosts work through "Unified Scaling Laws for Routed Language Models" (Clark et al., DeepMind 2022, arXiv 2202.01169) for the mixture-of-experts side and the DeepSeek sparse-attention paper (arXiv 2512.02556) for the attention side, treating Pope's blackboard framing on the podcast as the pedagogical lens. The episode separates what the papers establish from what Pope's practitioner intuition adds on top, with particular attention to how MoE on the FFN side and sparse attention on the QK side attack independent cost pools and can compound rather than compete.