MiniCPM: Small Language Models and Scalable Training Strategies

Hu, Tu, Han, He, Cui, Long, … Liu, Sun · Tsinghua University & Modelbest · arXiv v3, 3 Jun 2024
arXiv:2404.06395 1.2B & 2.4B non-embedding params WSD schedule · muP · 192 tok/param claim
Companion to the episode · interactive page link

The toolkit, and where the evidence is thin

Hover or tap a box. Border colour = how well the episode says it holds up.

Hover a box to see what the episode says about it.

The two released models

1.2Bnon-embedding params (GQA)
2.4Bnon-embedding params
~1.1Ttraining tokens
20Bexponential decay tokens
~6BSFT tokens

Reported relations on the paper's average score. The average mixes Chinese and English benchmarks and takes each model's better of two scoring methods.

Warmup-Stable-Decay: one stable run, many finished models

Relative learning rate vs. tokens. Each coloured branch is a short decay launched from a stable checkpoint. Decay shape is schematic.Illustrative curves

Why the loss drops at decay

Schematic loss shape only; the paper describes the drop (gradient cosine turns positive, curvature rises) but offers no theory.Illustrative · no paper data

Cost of a scaling study

Run count for k model sizes × k token budgets. Cosine needs a fresh run per pair; WSD branches from one run per size.Conceptual run counts

Where 192 tokens/param sits on the map

Log axis. Toggle layers; the cursor follows the slider below.

Same compute, different shape (C ≈ 6·N·D = 1e21 FLOPs)

Rectangle area is constant. Width is tokens D, height is parameters N. Values derived from C = 6ND.

The fit's exponents

Average-fit values as reported: α = 0.29, β = 0.23, η = (β−α)/(α+β) ≈ −0.10. The dashed "?" boxes mark uncertainty the paper does not report.

Wind tunnel: does the best learning rate hold still?

muP: reported best LR stays near 0.01 from 0.04B to 0.5B (also checked at 2.1B). The standard-parameterization curve is a drift sketch.SP curve illustrative

Batch-size rule on C4: bs = 1.21e9 / loss6.24

Plot of the reported fit. Lower loss calls for a bigger batch, but you need a trustworthy loss estimate first.

Reported results

Single runs, no error bars, proprietary data.

Claim-by-claim concern map

Hover a cell. Colour = concern level (blue low → red high).Editorial reading of the episode, not paper data

Hover a cell for the reason.

Stress-testing the 192

Step through why the number is shakier than the models.

References

  1. MiniCPM — Hu et al., 2024
  2. Textbooks Are All You Need (Phi-1) — Gunasekar et al., 2023
  3. TinyLlama — Zhang et al., 2024
  4. MobileLLM — Liu et al., 2024
  5. Small Language Models: Survey, Measurements, and Insights — Lu et al., 2024
  6. Understanding WSD: River Valley Loss Landscape — Wen et al., 2024
  7. Scaling Laws and Compute-Optimal Training Beyond Fixed Durations — Hägele et al., 2024
  8. The Road Less Scheduled — Defazio et al., 2024
  9. Scaling Vision Transformers — Zhai et al., 2022
  10. Tensor Programs V — Yang et al., 2022
  11. Tensor Programs VI — Yang et al., 2023
  12. Cerebras-GPT — Dey et al., 2023
  13. Scaling Exponents Across Parameterizations and Optimizers — Everett et al., 2024
  14. Chinchilla — Hoffmann et al., 2022
  15. Chinchilla Scaling: A Replication Attempt — Besiroglu et al., 2024
  16. Beyond Chinchilla-Optimal — Sardana & Frankle, 2023
  17. Small-scale proxies for large-scale Transformer instabilities — Wortsman et al., 2023
  18. Scaling Laws for Neural Language Models — Kaplan et al., 2020
  19. An Empirical Model of Large-Batch Training — McCandlish et al., 2018
  20. DeepSeek LLM — DeepSeek-AI, 2024