Craig W. Schmidt

How Much is Enough? The Diminishing Returns of Tokenization Training Data

2025 · arXiv (Cornell University)

Tokenization, a crucial initial step in natural language processing, is governed by several key parameters, such as the tokenization algorithm, vocabulary size, pre-tokenization strategy, inference strategy, and training data corpus. This paper investigates the impact …