Soham De

The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent

2019 · arXiv (Cornell University)

This paper studies how neural network architecture affects the speed of training. We introduce a simple concept called gradient confusion to help formally analyze this. When gradient confusion is high, stochastic gradients produced by different …