Jinho Park

How Do Large Language Models Acquire Factual Knowledge During Pretraining?

2024 · arXiv (Cornell University)

Despite the recent observation that large language models (LLMs) can store substantial factual knowledge, there is a limited understanding of the mechanisms of how they acquire factual knowledge through pretraining. This work addresses this gap …

How language models extrapolate outside the training data: A case study in Textualized Gridworld

2024 · arXiv (Cornell University)

Language models' ability to extrapolate learned behaviors to novel, more complex environments beyond their training scope is highly unknown. This study introduces a path planning task in a textualized Gridworld to probe language models' extrapolation …