Zhaoxuan Jiang

Post-Processing Tokenizer Adaptation Under a Fixed Vocabulary Budget: Compression–Performance Trade-offs Across Models and Tasks

2026 · Creativity and Innovation

Subword tokenization is a key component of modern language models, but general-purpose tokenizers are often inefficient for domain-specific text . Recent work has proposed post-processing replacement methods that adapt a tokenizer by replacing low-frequency subwords …