Armen Aghajanyan

Muppet: Massive Multi-task Representations with Pre-Finetuning

2021 · arXiv (Cornell University)

We propose pre-finetuning, an additional large-scale learning stage between language model pre-training and fine-tuning. Pre-finetuning is massively multi-task learning (around 50 datasets, over 4.8 million total labeled examples), and is designed to encourage learning of …