Gianluca Vico

Conditional Unigram Tokenization with Parallel Data

2025 · arXiv (Cornell University)

We introduce conditional unigram tokenization, a novel approach that extends unigram tokenization by conditioning target token probabilities on source-language tokens from parallel data. Given a fixed source tokenizer, our method learns a target tokenizer that …