Gradient Noise Scale Predicts Critical Batch Size for Large-Batch Training

An Empirical Model of Large-Batch Training — McCandlish, Kaplan, Amodei & the OpenAI Dota Team · 2018
arXiv:1812.06162 Interactive Visualization

References

  1. McCandlish et al., An Empirical Model of Large-Batch Training, 2018 — arXiv:1812.06162
  2. Shallue et al., Measuring the Effects of Data Parallelism on Neural Network Training, 2018 — arXiv:1811.03600
  3. Goyal et al., Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour, 2017 — arXiv:1706.02677
  4. You, Gitman, Ginsburg, Large Batch Training of Convolutional Networks (LARS), 2017 — arXiv:1708.03888
  5. Keskar et al., On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima, 2016 — arXiv:1609.04836
  6. Smith et al., Don't Decay the Learning Rate, Increase the Batch Size, 2017 — arXiv:1711.00489
  7. Kaplan et al., Scaling Laws for Neural Language Models, 2020 — arXiv:2001.08361
  8. Zhang et al., Which Algorithmic Choices Matter at Which Batch Sizes? Noisy Quadratic Model, 2019 — arXiv:1907.04164
  9. Jastrzębski et al., Three Factors Influencing Minima in SGD, 2017 — arXiv:1711.04623
  10. OpenAI, Dota 2 with Large Scale Deep Reinforcement Learning, 2019 — arXiv:1912.06680
  11. Balles, Romero, Hennig, adaptive sampling via gradient noise, 2016 — Scholar search