Alexis Conneau

Very Deep Convolutional Networks for Text Classification

2017

The dominant approach for many NLP tasks are recurrent neural networks, in particular LSTMs, and convolutional neural networks. However, these architectures are rather shallow in comparison to the deep convolutional networks which have pushed the …

Multilingual Speech Translation with Efficient Finetuning of Pretrained Models

2020 · arXiv (Cornell University)

We present a simple yet effective approach to build multilingual speech-to-text (ST) translation by efficient transfer learning from pretrained speech encoder and text decoder. Our key finding is that a minimalistic LNA (LayerNorm and Attention) …

mSLAM: Massively multilingual joint pre-training for speech and text

2022 · arXiv (Cornell University)

We present mSLAM, a multilingual Speech and LAnguage Model that learns cross-lingual cross-modal representations of speech and text by pre-training jointly on large amounts of unlabeled speech and text in multiple languages. mSLAM combines w2v-BERT …

Very Deep Convolutional Networks for Natural Language Processing.

2016 · arXiv (Cornell University)

The dominant approach for many NLP tasks are recurrent neural networks, in particular LSTMs, and convolutional neural networks. However, these architectures are rather shallow in comparison to the deep convolutional networks which are very successful …

Word Translation Without Parallel Data

2017 · arXiv (Cornell University)

State-of-the-art methods for learning cross-lingual word embeddings have relied on bilingual dictionaries or parallel corpora. Recent studies showed that the need for parallel data supervision can be alleviated with character-level information. While these methods showed …

Unsupervised Machine Translation Using Monolingual Corpora Only

2017 · arXiv (Cornell University)

Machine translation has recently achieved impressive performance thanks to recent advances in deep learning and the availability of large-scale parallel corpora. There have been numerous attempts to extend these successes to low-resource language pairs, yet …

SentEval: An Evaluation Toolkit for Universal Sentence Representations

2018 · arXiv (Cornell University)

We introduce SentEval, a toolkit for evaluating the quality of universal sentence representations. SentEval encompasses a variety of tasks, including binary and multi-class classification, natural language inference and sentence similarity. The set of tasks was …

What you can cram into a single vector: Probing sentence embeddings for\n linguistic properties

2018 · arXiv (Cornell University)

Although much effort has recently been devoted to training high-quality\nsentence embeddings, we still have a poor understanding of what they are\ncapturing. "Downstream" tasks, often based on sentence classification, are\ncommonly used to evaluate the quality of …

XNLI: Evaluating Cross-lingual Sentence Representations

2018 · arXiv (Cornell University)

Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, Veselin Stoyanov. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018.

Cross-lingual Language Model Pretraining

2019 · arXiv (Cornell University)

Recent studies have demonstrated the efficiency of generative pretraining for English natural language understanding. In this work, we extend this approach to multiple languages and show the effectiveness of cross-lingual pretraining. We propose two methods …

Phrase-Based & Neural Unsupervised Machine Translation

2018

Machine translation systems achieve near human-level performance on some languages, yet their effectiveness strongly relies on the availability of large amounts of parallel sentences, which hinders their applicability to the majority of language pairs. This …

Supervised Learning of Universal Sentence Representations from Natural\n Language Inference Data

2017 · arXiv (Cornell University)

Many modern NLP systems rely on word embeddings, previously trained in an\nunsupervised manner on large corpora, as base features. Efforts to obtain\nembeddings for larger chunks of text, such as sentences, have however not been\nso successful. …

What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties

2018

Although much effort has recently been devoted to training high-quality sentence embeddings, we still have a poor understanding of what they are capturing. "Downstream" tasks, often based on sentence classification, are commonly used to evaluate …

Unsupervised Cross-lingual Representation Learning at Scale

2020

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.

Emerging Cross-lingual Structure in Pretrained Language Models

2020

We study the problem of multilingual masked language modeling, i.e. the training of a single model on concatenated text from multiple languages, and present a detailed study of several factors that influence why these models …

XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale

2022 · Interspeech 2022

This paper presents XLS-R, a large-scale model for cross-lingual speech representation learning based on wav2vec 2.0.We train models with up to 2B parameters on nearly half a million hours of publicly available speech audio in …