Learning Distributed Representations for Multilingual Text Sequences

We propose a novel approach to learning dis-tributed representations of variable-length text sequences in multiple languages simultane-ously. Unlike previous work which often de-rive representations of multi-word sequences as weighted sums of individual word vec-tors, our model learns distributed representa-tions for phrases and sentences as a whole. Our work is similar in spirit to the recent paragraph vector approach but extends to the bilingual context so as to efficiently encode meaning-equivalent text sequences of multi-ple languages in the same semantic space. Our learned embeddings achieve state-of-the-art performance in the often used crosslingual document classification task (CLDC) with an accuracy of 92.7 for English to German and 91.5 for German to English. By learning text sequence representations as a whole, our model performs equally well in both classifi-cation directions in the CLDC task in which past work did not achieve. 1