1 · Flip the optimization problem
Toggle which question gets asked. Same loss law, different constraint.
2 · Tokens-per-parameter ruler
TPP = D / N. Log axis. Hover markers.
3 · Reported savings: FLOPs vs dollars Reported
Switch accounting. Values are the paper's reported/estimated numbers (model outputs, not trained models).
4 · Why dollars differ: utilization Reported
MFU assumed in the cost model. Output tokens are memory-bandwidth bound.
5 · Optimum explorer Illustrative
Real objective (6N·D + 2N·Dinf at fixed loss) using the published Hoffmann et al. constants (E=1.69, A=406.4, B=410.7, α=0.34, β=0.28). Target = 30B at 20 TPP. It does NOT reproduce the paper's 16.4B / 3.27T anchor; it shows the direction only.
6 · Coverage of the 47-model sweep Reported
Max TPP reached per size. Bright cell = reported maximum; dim = at or below it (not every cell is a separate run); empty = never run.
7 · Data exponent β depends on what you fit Reported
8 · Where evidence stops and recommendations start
Model size vs TPP, log–log. Toggle layers.
9 · Which way does the error push? Illustrative direction
Click a caveat. Arrows are qualitative; the paper did not quantify them.
References
- Sardana et al. — Beyond Chinchilla-Optimal (2024) — arXiv:2401.00448
- Hoffmann et al. — Training Compute-Optimal LLMs (2022) — arXiv:2203.15556
- Kaplan et al. — Scaling Laws for Neural Language Models (2020) — arXiv:2001.08361
- Touvron et al. — LLaMA (2023) — arXiv:2302.13971
- Touvron et al. — Llama 2 (2023) — arXiv:2307.09288; Llama Team — Llama 3 Herd (2024) — arXiv:2407.21783
- Muennighoff et al. — Scaling Data-Constrained LMs (2023) — arXiv:2305.16264
- Kumar et al. — Scaling Laws for Precision (2024) — arXiv:2411.04330
- Porian et al. — Resolving Discrepancies in Compute-Optimal Scaling (2024) — arXiv:2407.17467
- Pope et al. — Efficiently Scaling Transformer Inference (2022) — arXiv:2211.05102
- Kwon et al. — PagedAttention (2023) — arXiv:2309.06180
- Patel et al. — Splitwise (2024) — arXiv:2311.18677
- Gadre et al. — LMs scale reliably with over-training (2024) — arXiv:2403.08540
- Besiroglu et al. — Chinchilla Scaling: A replication attempt (2024) — arXiv:2404.10102
- de Vries — Go smol or go home (2023) — blog post, no arXiv
- Hägele et al. — Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations (2024) — arXiv:2405.18392
- Krajewski et al. — Scaling Laws for Fine-Grained MoE (2024) — arXiv:2402.07871
- Snell et al. — Scaling LLM Test-Time Compute Optimally (2024) — arXiv:2408.03314
- DeepSeek-AI — DeepSeek-V2 (2024) — arXiv:2405.04434; DeepSeek-V3 — arXiv:2412.19437