Qilin Li

AdaDiff: Adaptive Gradient Descent with the Differential of Gradient

2021 · Journal of Physics Conference Series

Abstract Optimization methods are crucial to train deep neural networks. Adaptive optimization methods, especially Adam, are wildly used because they aren’t sensitive to the selection of learning rate and converge fast. Recent work point out …