Nikos Karampatziakis

Meet in the Middle: A New Pre-training Paradigm

2023 · arXiv (Cornell University)

Most language models (LMs) are trained and applied in an autoregressive left-to-right fashion, assuming that the next token only depends on the preceding ones. However, this assumption ignores the potential benefits of using the full …