Scott Pesme

On Convergence-Diagnostic based Step Sizes for Stochastic Gradient Descent

2020 · arXiv (Cornell University)

Constant step-size Stochastic Gradient Descent exhibits two phases: a transient phase during which iterates make fast progress towards the optimum, followed by a stationary phase during which iterates oscillate around the optimal point. In this …