Hendra Setiawan

Accurate Knowledge Distillation with n-best Reranking

2023 · arXiv (Cornell University)

We propose utilizing n-best reranking to enhance Sequence-Level Knowledge Distillation (Kim and Rush, 2016) where we extract pseudo-labels for student model's training data from top n-best hypotheses and leverage a diverse set of models with …

One Wide Feedforward is All You Need

2023 · arXiv (Cornell University)

The Transformer architecture has two main non-embedding components: Attention and the Feed Forward Network (FFN). Attention captures interdependencies between words regardless of their position, while the FFN non-linearly transforms each input token independently. In this …