Maxat Tezekbayev

Speeding Up Entmax

2021 · arXiv (Cornell University)

Softmax is the de facto standard in modern neural networks for language processing when it comes to normalizing logits. However, by producing a dense probability distribution each token in the vocabulary has a nonzero chance …