Sachin Kumar

FLEXITOKENS: Flexible Tokenization for Evolving Language Models

2026

Adapting language models to new data distributions by simple finetuning is challenging.This is due to the rigidity of their subword tokenizers, which typically remain unchanged during adaptation.This inflexibility often leads to inefficient tokenization, causing overfragmentation …