Yuhai Tu

The inverse variance–flatness relation in stochastic gradient descent is critical for finding flat minima

2021 · Proceedings of the National Academy of Sciences

Despite tremendous success of the stochastic gradient descent (SGD) algorithm in deep learning, little is known about how SGD finds generalizable solutions at flat minima of the loss function in high-dimensional weight space. Here, we …