Podcast companion · Hal Turing & Dr. Ada Shannon

Scaling Laws for Precision: Bits, Parameters, and Training Tokens

Kumar, Ankner, Spector, Bordelon, Muennighoff, Paul, Pehlevan, Ré, Raghunathan · Nov 2024
arXiv 2411.04330

Three regimes, one discount

Click a box to see what it covers.

Tokens per parameter (log scale)

Values as stated in the episode: ~20 Chinchilla, ~1,000 the paper's runs, ~2,000 Llama-3-8B, ~100,000 extrapolated.

PTQ damage vs data Illustrative

Paper: loss increase from rounding is a power of D/N, exponential as bits drop, smaller for bigger N. Shapes only; constants here are invented for display.

Penalty heatmap: bits × D/N Illustrative

Hover a cell.
Hatched columns lie beyond the ~1,000 D/N the paper's runs reach: power-law extrapolation only.

Effective parameters: N·(1 − e^(−P/γ)) Illustrative γ

Paper's weight-only form. γ below is a slider, not a fitted value.

Bits vs parameters trade Illustrative

Real parameter count needed to hold N_eff = 1 (relative units) at each bit width, same γ as above.

Fixed compute, lower precision: C ≈ 6·N·D·P/16

Exact cost relation from the paper; axes are relative (log2), no measured values.

Cost model vs hardware Illustrative

Episode: H100 FP8 ≈ 2× BF16; FP6 has no native tensor-core path; INT7 is not a speed class. Step view is a sketch of that argument, untested in the paper.

Model size needed to hold loss (as stated in episode)

Claim vs evidence

Hover a cell.

References

  1. Scaling Laws for Precision — Kumar et al., 2024
  2. Scaling Laws for Neural Language Models — Kaplan et al., 2020
  3. Training Compute-Optimal LLMs (Chinchilla) — Hoffmann et al., 2022
  4. FP8 Formats for Deep Learning — Micikevicius et al., 2022
  5. BitNet b1.58, The Era of 1-bit LLMs — Ma et al., 2024
  6. The case for 4-bit precision: k-bit Inference Scaling Laws — Dettmers & Zettlemoyer, 2023
  7. Language models scale reliably with over-training — Gadre et al., 2024
  8. Beyond Chinchilla-Optimal — Sardana & Frankle, 2023
  9. GPTQ — Frantar et al., 2022
  10. Physics of LMs 3.3, Knowledge Capacity Scaling Laws — Allen-Zhu & Li, 2024
  11. Pretraining LLMs with NVFP4 — NVIDIA, 2025
  12. Scaling Laws for Floating Point Quantization Training — Sun et al., 2025
  13. Scaling Law for Quantization-Aware Training — Chen et al., 2025
  14. QuaRot / SpinQuant — Ashkboos et al.; Liu et al., 2024