Kaichao You

Learning Stages: Phenomenon, Root Cause, Mechanism Hypothesis, and Implications.

2019 · arXiv (Cornell University)

Learning rate decay (lrDecay) is a \emph{de facto} technique for training modern neural networks. It starts with a large learning rate and then decays it multiple times. It is empirically observed to help both optimization …