Nikunj Saunshi

A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks

2020 · arXiv (Cornell University)

Autoregressive language models, pretrained using large text corpora to do well on next word prediction, have been successful at solving many downstream tasks, even with zero-shot usage. However, there is little theoretical understanding of this …