Zhoufutu Wen

Quantification of Large Language Model Distillation

2025 · arXiv (Cornell University)

Model distillation is a fundamental technique in building large language models (LLMs), transferring knowledge from a teacher model to a student model. However, distillation can lead to model homogenization, reducing diversity among models and impairing …