Ruiqi Wang

Divergence Results and Convergence of a Variance Reduced Version of ADAM

2022 · arXiv (Cornell University)

Stochastic optimization algorithms using exponential moving averages of the past gradients, such as ADAM, RMSProp and AdaGrad, have been having great successes in many applications, especially in training deep neural networks. ADAM in particular stands …