Scaling Inference Demand into Chinchilla: Training Smaller Models Longer

Companion to "Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws" — Sardana, Portes, Doubov, Frankle (Databricks MosaicML) · ICML 2024
arXiv 2401.00448 47 trained models · 150M–6B 10 – 10,000 tokens/param Weak spot: extreme end = one small model

1 · Flip the optimization problem

Toggle which question gets asked. Same loss law, different constraint.

2 · Tokens-per-parameter ruler

TPP = D / N. Log axis. Hover markers.

Hover a marker.

3 · Reported savings: FLOPs vs dollars Reported

Switch accounting. Values are the paper's reported/estimated numbers (model outputs, not trained models).

4 · Why dollars differ: utilization Reported

MFU assumed in the cost model. Output tokens are memory-bandwidth bound.

5 · Optimum explorer Illustrative

Real objective (6N·D + 2N·Dinf at fixed loss) using the published Hoffmann et al. constants (E=1.69, A=406.4, B=410.7, α=0.34, β=0.28). Target = 30B at 20 TPP. It does NOT reproduce the paper's 16.4B / 3.27T anchor; it shows the direction only.

6 · Coverage of the 47-model sweep Reported

Max TPP reached per size. Bright cell = reported maximum; dim = at or below it (not every cell is a separate run); empty = never run.

Hover a cell.

7 · Data exponent β depends on what you fit Reported

8 · Where evidence stops and recommendations start

Model size vs TPP, log–log. Toggle layers.

Hover a point.

9 · Which way does the error push? Illustrative direction

Click a caveat. Arrows are qualitative; the paper did not quantify them.

Click a row.

References

  1. Sardana et al. — Beyond Chinchilla-Optimal (2024) — arXiv:2401.00448
  2. Hoffmann et al. — Training Compute-Optimal LLMs (2022) — arXiv:2203.15556
  3. Kaplan et al. — Scaling Laws for Neural Language Models (2020) — arXiv:2001.08361
  4. Touvron et al. — LLaMA (2023) — arXiv:2302.13971
  5. Touvron et al. — Llama 2 (2023) — arXiv:2307.09288; Llama Team — Llama 3 Herd (2024) — arXiv:2407.21783
  6. Muennighoff et al. — Scaling Data-Constrained LMs (2023) — arXiv:2305.16264
  7. Kumar et al. — Scaling Laws for Precision (2024) — arXiv:2411.04330
  8. Porian et al. — Resolving Discrepancies in Compute-Optimal Scaling (2024) — arXiv:2407.17467
  9. Pope et al. — Efficiently Scaling Transformer Inference (2022) — arXiv:2211.05102
  10. Kwon et al. — PagedAttention (2023) — arXiv:2309.06180
  11. Patel et al. — Splitwise (2024) — arXiv:2311.18677
  12. Gadre et al. — LMs scale reliably with over-training (2024) — arXiv:2403.08540
  13. Besiroglu et al. — Chinchilla Scaling: A replication attempt (2024) — arXiv:2404.10102
  14. de Vries — Go smol or go home (2023) — blog post, no arXiv
  15. Hägele et al. — Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations (2024) — arXiv:2405.18392
  16. Krajewski et al. — Scaling Laws for Fine-Grained MoE (2024) — arXiv:2402.07871
  17. Snell et al. — Scaling LLM Test-Time Compute Optimally (2024) — arXiv:2408.03314
  18. DeepSeek-AI — DeepSeek-V2 (2024) — arXiv:2405.04434; DeepSeek-V3 — arXiv:2412.19437