CoRe-KD: Contrastive Multi-Teacher and Retention-Aware Knowledge Distillation for Continual Multilingual Neural Machine Translation

A longstanding objective of multilingual neural machine translation (MNMT) is to expand models to acquire new knowledge from incremental translation tasks over time. Existing methods predominantly concentrate on mitigating catastrophic forgetting by compromising between original and new language pairs, leading to suboptimal performance on both translation tasks. In this work, we propose CoRe-KD, a novel contrastive multi-teacher and retention-aware knowledge distillation method to prevent forgetting. Specifically, CoRe-KD introduces a contrastive multiteacher distillation strategy that assesses the representational relevance of each teacher model to the student model via contrastive learning, and a retention mechanism that allows the student model to mimic outputs from previous state during training. Experimental results demonstrate that CoRe-KD effectively adapts to new translation directions while preserving the performance of original tasks.