Lecheng Wang

Theoretical Proof that Auto-regressive Language Models Collapse when Real-world Data is a Finite Set

2024 · arXiv (Cornell University)

Auto-regressive language models (LMs) have been widely used to generate data in data-scarce domains to train new LMs, compensating for the scarcity of real-world data. Previous work experimentally found that LMs collapse when trained on …