Seonghyeon Ye

How Do Large Language Models Acquire Factual Knowledge During Pretraining?

2024 · arXiv (Cornell University)

Despite the recent observation that large language models (LLMs) can store substantial factual knowledge, there is a limited understanding of the mechanisms of how they acquire factual knowledge through pretraining. This work addresses this gap …