Sanjeev Arora

A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks

2020 · arXiv (Cornell University)

Autoregressive language models, pretrained using large text corpora to do well on next word prediction, have been successful at solving many downstream tasks, even with zero-shot usage. However, there is little theoretical understanding of this …

A Latent Variable Model Approach to PMI-based Word Embeddings

2016 · Transactions of the Association for Computational Linguistics

Semantic word embeddings represent the meaning of a word via a vector, and are created by diverse methods. Many use nonlinear operations on co-occurrence statistics, and have hand-tuned hyperparameters and reweighting methods. This paper proposes …

A Simple but Tough-to-Beat Baseline for Sentence Embeddings

2017 · International Conference on Learning Representations

The success of neural network methods for computing word embeddings has motivated methods for generating semantic embeddings of longer pieces of text, such as sentences and paragraphs. Surprisingly, Wieting et al (ICLR'16) showed that such …