Haoyu Sheng

Knowledge Distillation for Recurrent Neural Network Language Modeling with Trust Regularization

2019

Recurrent Neural Networks (RNNs) have dominated language modeling because of their superior performance over traditional N-gram based models. In many applications, a large Recurrent Neural Network language model (RNNLM) or an ensemble of several RNNLMs …