Yangyang Shi

Knowledge Distillation for Recurrent Neural Network Language Modeling with Trust Regularization

2019

Recurrent Neural Networks (RNNs) have dominated language modeling because of their superior performance over traditional N-gram based models. In many applications, a large Recurrent Neural Network language model (RNNLM) or an ensemble of several RNNLMs …

Scaling Parameter-Constrained Language Models with Quality Data

2024 · arXiv (Cornell University)

Scaling laws in language modeling traditionally quantify training loss as a function of dataset size and model parameters, providing compute-optimal estimates but often neglecting the impact of data quality on model generalization. In this paper, …