Jonathan Patsenker

Exponential weight averaging as damped harmonic motion

2023 · arXiv (Cornell University)

The exponential moving average (EMA) is a commonly used statistic for providing stable estimates of stochastic quantities in deep learning optimization. Recently, EMA has seen considerable use in generative models, where it is computed with …