Toan Nguyen

Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation

2017 · arXiv (Cornell University)

We present a simple method to improve neural translation of a low-resource language pair using parallel data from a related, also low-resource, language pair. The method is based on the transfer method of Zoph et …

Transformers without Tears: Improving the Normalization of Self-Attention

2019 · arXiv (Cornell University)

We evaluate three simple, normalization-centric changes to improve Transformer training. First, we show that pre-norm residual connections (PreNorm) and smaller initializations enable warmup-free, validation-based training with large learning rates. Second, we propose $\ell_2$ normalization with …

Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets

2022 · Transactions of the Association for Computational Linguistics

Abstract With the success of large-scale pre-training and multilingual modeling in Natural Language Processing (NLP), recent years have seen a proliferation of large, Web-mined text datasets covering hundreds of languages. We manually audit the quality …