Paul Caillon

Growing Neural Networks have Flat Optima and Generalize Better

2024 · HAL (Le Centre pour la Communication Scientifique Directe)

In this work, we study the loss landscape of growing neural networks and show that they have flatter minima than when trained with all of their parameters from random initialization. Then, we further evaluate and …