Yao Hu

Towards the Law of Capacity Gap in Distilling Language Models

2025

Language model (LM) distillation aims at distilling the knowledge in a large teacher LM to a small student one. As a critical issue facing LM distillation, a superior student often arises from a teacher of …