Ambroise Odonnat

A Mechanistic Study of Transformers Training Dynamics

2024 · arXiv (Cornell University)

Large-scale pretraining of transformers has been central to the success of foundation models. However, the scale of those models limits our understanding of the mechanisms at play during optimization. In this work, we study the …