Fangyu Zou

Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Acceleration

2021 · arXiv (Cornell University)

Adam is one of the most influential adaptive stochastic algorithms for training deep neural networks, which has been pointed out to be divergent even in the simple convex setting via a few simple counterexamples. Many …