Yuanxin Liu

Marginal Utility Diminishes: Exploring the Minimum Knowledge for BERT Knowledge Distillation

2021 · arXiv (Cornell University)

Recently, knowledge distillation (KD) has shown great success in BERT compression. Instead of only learning from the teacher's soft label as in conventional KD, researchers find that the rich information contained in the hidden layers …

DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models

2024 · arXiv (Cornell University)

The visual projector, which bridges the vision and language modalities and facilitates cross-modal alignment, serves as a crucial component in MLLMs. However, measuring the effectiveness of projectors in vision-language alignment remains under-explored, which currently can …