William Fedus

Emergent Abilities of Large Language Models

2022 · arXiv (Cornell University)

Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictable phenomenon that we refer to as emergent abilities …

MaskGAN: Better Text Generation via Filling in the______

2018 · arXiv (Cornell University)

Neural text generation models are often autoregressive language models or seq2seq models. These models generate text by sampling words sequentially, with each word conditioned on the previous word, and are state-of-the-art for several machine translation …

Deep Graph Infomax

2018 · Apollo (University of Cambridge)

We present Deep Graph Infomax (DGI), a general approach for learning node representations within graph-structured data in an unsupervised manner. DGI relies on maximizing mutual information between patch representations and corresponding high-level summaries of graphs---both …

Language GANs Falling Short

2020 · International Conference on Learning Representations

Traditional natural language generation (NLG) models are trained using maximum likelihood estimation (MLE) which differs from the sample generation inference procedure. During training the ground truth tokens are passed to the model, however, during inference, …

Scaling Instruction-Finetuned Language Models

2022 · arXiv (Cornell University)

Finetuning language models on a collection of datasets phrased as instructions has been shown to improve model performance and generalization to unseen tasks. In this paper we explore instruction finetuning with a particular focus on …