Avihu Dekel

Exploring the Benefits of Tokenization of Discrete Acoustic Units

2024 · arXiv (Cornell University)

Tokenization algorithms that merge the units of a base vocabulary into larger, variable-rate units have become standard in natural language processing tasks. This idea, however, has been mostly overlooked when the vocabulary consists of phonemes …