I. Babuschkin

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

2022 · arXiv (Cornell University)

In this paper we propose to study generalization of neural networks on small algorithmically generated datasets. In this setting, questions about data efficiency, memorization, generalization, and speed of learning can be studied in great detail. …

Scaling Language Models: Methods, Analysis & Insights from Training Gopher

2021 · arXiv (Cornell University)

Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world. In this paper, we present an analysis of Transformer-based language model …