Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient

Switch-style top-1 MoE layer

Experts have the same width as the dense feed-forward layer. Step through one token's path.

Total vs active parameters reported

Same 1.1B in memory. Compute follows the active number.

Router scores, 10 tokens × 4 experts Illustrative

Seeded random scores, not model output. Outlined cell = top-1 choice. Hover a cell.

How the joint law is assembled

Structure as described in the episode; hover a block. Full fitted formula is in the paper.

Data exponent shifts with expert count endpoints reported

≈ −0.197 at E=1, ≈ −0.265 at E=32. Line between them is a straight-line guide, Illustrative.

Compute-optimal point at 10²² FLOPs reported

More experts → fewer active parameters, more tokens. At 10²⁰ FLOPs an MoE can match the compute-optimal dense model with 60% of the compute.

E-hat saturating transform schematic

Log-scaled axis marking the reported start ≈ 2 and ceiling ≈ 290. Curve shape not drawn.

Memory-matched comparison, 1.1B total reported + derived

Dense 1.1B on 8B tokens vs MoEs with the same total parameters trained on E× tokens.

Why the rule stops at E = 8

Tokens-per-parameter ratio, log axis. E=16 lands near 414, outside the trained range.

What “memory” counts

Parameters and KV cache. Hover blocks.

Per-token cost and cache

Experts touched in one decode step Illustrative

Assumes each token independently picks a uniformly random expert: fraction touched = 1 − (1 − 1/E)^B. Not measured in the paper.

Fit error vs the head-to-head gap

Fit errors reported. Gap not reported; the hatched bar is the host's rough read of the plot ("a few hundredths of a nat").

Scope check

Verdict on the direct test

1.1B dense vs 2-expert (709M active) and 4-expert (426M active), same FLOPs. The MoEs win over a bounded interval. Plausible, not established: no seeds, appendix lists each configuration once.