Jianhui Sun

Demystify Hyperparameters for Stochastic Optimization with Transferable Representations

2022 · Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

This paper studies the convergence and generalization of a large class of Stochastic Gradient Descent (SGD) momentum schemes, in both learning from scratch and transferring representations with fine-tuning. Momentum-based acceleration of SGD is the default …