Nolan Miller

Training trajectories, mini-batch losses and the curious role of the learning rate

2023 · arXiv (Cornell University)

Stochastic gradient descent plays a fundamental role in nearly all applications of deep learning. However its ability to converge to a global minimum remains shrouded in mystery. In this paper we propose to study the …