Soup-of-Experts: Pretraining Specialists via Parameter Averaging

Pierre Ablin, Angelos Katharopoulos, Skyler Seto, David Grangier (Apple) · arXiv, Feb 2025
arXiv:2502.01804 Interactive Visualization

Pay once vs. pay per target

Top: CRISP trains a fresh model for every target. Bottom: the soup trains one bank, then merges. Schematic

Three ideas in the background

Hover a card. Soups, hypernetworks and domain experts, as framed in the episode.

4096 clusters = 64 × 64

RedPajama-v2 split by hierarchical k-means on sentence-BERT embeddings. A mixture is a histogram over these cells. Illustrative histogram

clusters per histogram s =4
Hover a cell.

Meta-distribution over s

Reported ablation findings, drawn as qualitative cards (no numeric values reported in the episode).

One training step

Step through the loop. Gradients reach S, all experts and the MLP together.

Mixing recipe: S + Σ αᵢ Eᵢ

Pick a target. Cells show 128 expert coefficients. Illustrative α values

Hover an expert.

Hypernetwork? Only partly

The reachable weights are the span of S and 128 experts, not arbitrary.

Training budget

Reported token counts; totals marked derived are arithmetic from them. Reported

Specialized loss: qualitative ordering

Lower is better. Order only; the paper text gives no end-point numbers and no error bars. Illustrative

How many target documents?

Log scale. Reported: ~3 docs match the generic model on average; 100–1000 reach the soup's best. Reported

Ablations

Hover for the reported finding.

Amortization break-even

Derived in the episode: soup ≈ 174B generic-equivalent tokens (134B ÷ 0.77). CRISP = 17B per target, or 134B at matched tokens. Cost is in billions of tokens.

targets N =16

What the headline comparison does and doesn't show

Status of each check. Hover a row.

References

  1. Ablin et al., Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging, 2025.
  2. Li et al., Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models, 2022.
  3. Gururangan et al., Scaling Expert Language Models with Unsupervised Domain Discovery (c-BTM), 2023.
  4. Ha, Dai, Le, Hypernetworks, 2017.
  5. HyperFormer / Hyper-X / HyperTuning line (Mahabadi; Üstün; Phang), 2021–2023.
  6. Puigcerver et al., From Sparse to Soft Mixtures of Experts, 2024.
  7. Zhong et al., Lory: Fully Differentiable MoE for Autoregressive LM Pre-training, 2024.
  8. Xia et al., Sheared LLaMA, 2023.
  9. Xie et al., DoReMi: Optimizing Data Mixtures Speeds Up LM Pre-training, 2023.
  10. Ilharco et al.; Yadav et al.; Yu et al., Task Arithmetic / TIES-Merging / DARE.
  11. Gadre et al., Language Models Scale Reliably With Over-Training and on Downstream Tasks, 2024.
  12. Model merging surveys, 2024–2025.
  13. Also cited in the episode: Wortsman et al., Model soups (ICML 2022); Grangier et al., CRISP clustered-importance sampling (2024); Gross et al., 2017.