Shangquan Sun

Logit Standardization in Knowledge Distillation

2024 · arXiv (Cornell University)

Knowledge distillation involves transferring soft labels from a teacher to a student using a shared temperature-based softmax function. However, the assumption of a shared temperature between teacher and student implies a mandatory exact match between …