Dense block vs MoE block
MoE swaps the feed-forward layer for many experts plus a router. Schematic
Active vs total parameters
Example from the episode: a "64×3M" model. Reported
Who said what about the MoE advantage?
Curve shapes show direction of claims only, not measured values. Illustrative
Granularity G: slice the experts thinner
G = dense FFN width ÷ expert width. Experts = G × (base count), top-k scales with G. Base here: top-2 of 8, as in the episode. Schematic layout
G = 1
Routing combinations per token
C(8G, 2G), computed exactly. G=1 → 28; G=4 → ≈10.5 million (both quoted in the episode).
How the law was assembled
Click a step. Color shows what backs it: data, a plot, or argument.
The fitted form
Hover each term. Equation as described in the episode.
Exponents: MoE vs dense
Benefit of G vs routing cost
Toy model: net = (r/Gγ + (1−r)) + λ·G. The shape shows why the optimum depends on γ and routing cost; r and λ are free sliders, not fitted values. Illustrative
Compute-optimal tokens (10th–90th pct bootstrap)
Resample 80% of points × 100 refits. Log axis. Reported
Refit coefficients: E=64 vs E=16
Hover bars. E=64 γ and β from the main fit; E=16 from Appendix D. Reported
Optimal granularity climbs with budget
Three reported anchor points; the 1T row is from solving the fitted law. Reported
Dense compute needed to match one compute-optimal MoE
Multipliers were read off a plot, not tabulated. Reported (approx.)
Fit error
RMSE in loss; losses sit roughly in 2.8–3.8. The holdout withholds the best 20% of runs, not the largest.
How far does the extrapolation reach?
Trained range vs the headline claims. FLOPs are the episode's back-of-envelope: 6 × active params × tokens.
Evidence matrix
Where does each claim come from? Hover a row.
The ragged training grid
Only two cells are stated: G=16 stops at the 49M model; the 85M model runs G≤4 only. Other cells assumed filled. Coverage sketch
Where granularity hit a ceiling
References
- Krajewski et al., Scaling Laws for Fine-Grained Mixture of Experts, 2024
- Dai et al., DeepSeekMoE, 2024
- Clark et al., Unified Scaling Laws for Routed Language Models, 2022
- Artetxe et al., Efficient Large Scale Language Modeling with Mixtures of Experts, 2022
- He, Mixture of A Million Experts (PEER), 2024
- Kaplan et al., Scaling Laws for Neural Language Models, 2020
- Hoffmann et al., Training Compute-Optimal LLMs (Chinchilla), 2022
- Zhou et al., Mixture-of-Experts with Expert Choice Routing, 2022
- Ludziejewski et al., Joint MoE Scaling Laws, 2025
- Fedus et al., Switch Transformers, 2021
- Lepikhin et al., GShard, 2020
- Jiang et al., Mixtral of Experts, 2024
- Geva et al., FFN Layers Are Key-Value Memories, 2021
- Frantar et al., Scaling Laws for Sparsely-Connected Foundation Models, 2023
- Liew et al., Scaling Laws for Upcycling MoE LMs, 2025