Alexandre Variengien

How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model

2023 · arXiv (Cornell University)

Pre-trained language models can be surprisingly adept at tasks they were not explicitly trained on, but how they implement these capabilities is poorly understood. In this paper, we investigate the basic mathematical abilities often acquired …