Rendi Chevi

Which Student is Best? A Comprehensive Knowledge Distillation Exam for Task-Specific BERT Models

2022 · arXiv (Cornell University)

We perform knowledge distillation (KD) benchmark from task-specific BERT-base teacher models to various student models: BiLSTM, CNN, BERT-Tiny, BERT-Mini, and BERT-Small. Our experiment involves 12 datasets grouped in two tasks: text classification and sequence labeling …