Sangwon Yu

Rare Words Degenerate All Words.

2021 · arXiv (Cornell University)

Despite advances in neural network language model, the representation degeneration problem of embeddings is still challenging. Recent studies have found that the learned output embeddings are degenerated into a narrow-cone distribution which makes the similarity …