Gabriel Pereyra

Large scale distributed neural network training through online\n distillation

2018 · arXiv (Cornell University)

Techniques such as ensembling and distillation promise model quality\nimprovements when paired with almost any base model. However, due to increased\ntest-time cost (for ensembles) and increased complexity of the training\npipeline (for distillation), these techniques are challenging …

OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization

2022 · arXiv (Cornell University)

Recent work has shown that fine-tuning large pre-trained language models on a collection of tasks described via instructions, a.k.a. instruction-tuning, improves their zero and few-shot generalization to unseen tasks. However, there is a limited understanding …