Dwarak Govind Parthiban

On the Softmax Bottleneck of Recurrent Language Models

2021 · Proceedings of the AAAI Conference on Artificial Intelligence

Recent research has pointed to a limitation of word-level neural language models with softmax outputs. This limitation, known as the softmax bottleneck refers to the inability of these models to produce high-rank log probability (log …