Harsh Mehta

Extreme Memorization via Scale of Initialization

2021 · arXiv (Cornell University)

We construct an experimental setup in which changing the scale of initialization strongly impacts the implicit regularization induced by SGD, interpolating from good generalization performance to completely memorizing the training set while making little progress …