Scaling Laws for Fine-Grained Mixture of Experts

Krajewski, Ludziejewski, Adamczewski, Pióro, Krutul, Antoniak, Ciebiera, Król, Odrzygóźdź, Sankowski, Cygan, Jaszczur · 2024 · Univ. of Warsaw, IDEAS NCBR et al. · early arXiv version (Feb 2024)
arXiv:2402.07871 Interactive page

Dense block vs MoE block

MoE swaps the feed-forward layer for many experts plus a router. Schematic

Active vs total parameters

Example from the episode: a "64×3M" model. Reported

Who said what about the MoE advantage?

Curve shapes show direction of claims only, not measured values. Illustrative

Granularity G: slice the experts thinner

G = dense FFN width ÷ expert width. Experts = G × (base count), top-k scales with G. Base here: top-2 of 8, as in the episode. Schematic layout
G = 1

Routing combinations per token

C(8G, 2G), computed exactly. G=1 → 28; G=4 → ≈10.5 million (both quoted in the episode).

How the law was assembled

Click a step. Color shows what backs it: data, a plot, or argument.

The fitted form

Hover each term. Equation as described in the episode.

Exponents: MoE vs dense

Benefit of G vs routing cost

Toy model: net = (r/Gγ + (1−r)) + λ·G. The shape shows why the optimum depends on γ and routing cost; r and λ are free sliders, not fitted values. Illustrative

Compute-optimal tokens (10th–90th pct bootstrap)

Resample 80% of points × 100 refits. Log axis. Reported

Refit coefficients: E=64 vs E=16

Hover bars. E=64 γ and β from the main fit; E=16 from Appendix D. Reported

Optimal granularity climbs with budget

Three reported anchor points; the 1T row is from solving the fitted law. Reported

Dense compute needed to match one compute-optimal MoE

Multipliers were read off a plot, not tabulated. Reported (approx.)

Fit error

RMSE in loss; losses sit roughly in 2.8–3.8. The holdout withholds the best 20% of runs, not the largest.

How far does the extrapolation reach?

Trained range vs the headline claims. FLOPs are the episode's back-of-envelope: 6 × active params × tokens.

Evidence matrix

Where does each claim come from? Hover a row.

The ragged training grid

Only two cells are stated: G=16 stops at the 49M model; the 85M model runs G≤4 only. Other cells assumed filled. Coverage sketch

Where granularity hit a ceiling

References

  1. Krajewski et al., Scaling Laws for Fine-Grained Mixture of Experts, 2024
  2. Dai et al., DeepSeekMoE, 2024
  3. Clark et al., Unified Scaling Laws for Routed Language Models, 2022
  4. Artetxe et al., Efficient Large Scale Language Modeling with Mixtures of Experts, 2022
  5. He, Mixture of A Million Experts (PEER), 2024
  6. Kaplan et al., Scaling Laws for Neural Language Models, 2020
  7. Hoffmann et al., Training Compute-Optimal LLMs (Chinchilla), 2022
  8. Zhou et al., Mixture-of-Experts with Expert Choice Routing, 2022
  9. Ludziejewski et al., Joint MoE Scaling Laws, 2025
  10. Fedus et al., Switch Transformers, 2021
  11. Lepikhin et al., GShard, 2020
  12. Jiang et al., Mixtral of Experts, 2024
  13. Geva et al., FFN Layers Are Key-Value Memories, 2021
  14. Frantar et al., Scaling Laws for Sparsely-Connected Foundation Models, 2023
  15. Liew et al., Scaling Laws for Upcycling MoE LMs, 2025