Distilling Knowledge for Distant Speech Recognition via Parallel Data

In order to improve the performance of distant speech recognition tasks, this paper proposes to distill knowledge from the close-talking model to the distant model using parallel data. The close-talking model is called the teacher model. The distant model is called the student model. The student model is trained to imitate the output distributions of the teacher model. This constraint can be realized by minimizing the Kullback-Leibler (KL) divergence between the output distribution of the student model and the teacher model. Experimental results on AMI datasets show that the best student model achieves up to 8.5% relative word error rate (WER) reduction when compared with the conventionally-trained baseline models.