Nasser Zalmout

Optimizing Tokenization Choice for Machine Translation across Multiple Target Languages

2017 · ˜The œPrague Bulletin of Mathematical Linguistics

Abstract Tokenization is very helpful for Statistical Machine Translation (SMT), especially when translating from morphologically rich languages. Typically, a single tokenization scheme is applied to the entire source-language text and regardless of the target language. …