Puneeth Salad

Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher

2021 · arXiv (Cornell University)

With ever growing scale of neural models, knowledge distillation (KD) attracts more attention as a prominent tool for neural model compression. However, there are counter intuitive observations in the literature showing some challenging limitations of …