Tran, Nguyen-Hoang

Revisiting LARS for Large Batch Training Generalization of Neural Networks

2023 · arXiv (Cornell University)

This paper explores Large Batch Training techniques using layer-wise adaptive scaling ratio (LARS) across diverse settings, uncovering insights. LARS algorithms with warm-up tend to be trapped in sharp minimizers early on due to redundant ratio …