Kundan Krishna

Downstream Datasets Make Surprisingly Good Pretraining Corpora

2023

For most natural language processing tasks, the dominant practice is to finetune large pretrained transformer models (e.g., BERT) using smaller downstream datasets.Despite the success of this approach, it remains unclear to what extent these gainsare …