Diego Granziol

Learning Rates as a Function of Batch Size: A Random Matrix Theory\n Approach to Neural Network Training

2020 · arXiv (Cornell University)

We study the effect of mini-batching on the loss landscape of deep neural\nnetworks using spiked, field-dependent random matrix theory. We demonstrate\nthat the magnitude of the extremal values of the batch Hessian are larger than\nthose of …