4-bit Precision Beats 8-bit: k-bit Inference Scaling Laws

Dettmers & Zettlemoyer · University of Washington · arXiv Dec 2022 (rev. Feb 2023) · ICML 2023 · >35,000 experiments · OPT, BLOOM(Z), Pythia/NeoX, GPT-2 · 19M–176B params
arXiv:2212.09720

References

  1. The case for 4-bit precision: k-bit Inference Scaling Laws (2212.09720)
  2. GPT-2: Language Models are Unsupervised Multitask Learners (Radford et al., 2019)
  3. Language Models are Few-Shot Learners (2005.14165)
  4. Scaling Laws for Neural Language Models (2001.08361)
  5. LLM.int8() (2208.07339)
  6. GPTQ (2210.17323)
  7. AWQ (2306.00978)
  8. SmoothQuant (2211.10438)
  9. QLoRA (NF4) (2305.14314)
  10. QuIP (2307.13304) / QuIP# (2402.04396)
  11. AQLM (2401.06118)
  12. Scaling Laws for Precision (2411.04330)
  13. Efficiently Scaling Transformer Inference (2211.05102)
  14. Chinchilla (2203.15556)
  15. FP8 Formats for Deep Learning (2209.05433)
  16. KIVI (2402.02750) / KVQuant (2401.18079)