Sid Black

The Pile: An 800GB Dataset of Diverse Text for Language Modeling

2020 · arXiv (Cornell University)

Recent work has demonstrated that increased training dataset diversity improves general cross-domain knowledge and downstream generalization capability for large-scale language models. With this in mind, we present \textit{the Pile}: an 825 GiB English text corpus …