Benjamin Frey

Systematic Analysis of Tokenization Properties in Low-Resource Polysynthetic NMT

2026 · Proceedings of the ... International Florida Artificial Intelligence Research Society Conference

Tokenization is critical for neural machine translation (NMT), especially for polysynthetic and low-resource languages with complex morphology. While subword methods like Byte Pair Encoding (BPE) are common in modern NMT, their effectiveness varies across languages, …