Xucheng Ye

RoSearch: Search for Robust Student Architectures When Distilling Pre-trained Language Models

2021 · arXiv (Cornell University)

Pre-trained language models achieve outstanding performance in NLP tasks. Various knowledge distillation methods have been proposed to reduce the heavy computation and storage requirements of pre-trained language models. However, from our observations, student models acquired …