Urvashi Khandelwal

Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context

2018

We know very little about how neural language models (LM) use prior linguistic context.In this paper, we investigate the role of context in an LSTM LM, through ablation studies.Specifically, we analyze the increase in perplexity …

Emergent linguistic structure in artificial neural networks trained by self-supervision

2020 · Proceedings of the National Academy of Sciences

This paper explores the knowledge of linguistic structure learned by large artificial neural networks, trained via self-supervision, whereby the model simply tries to predict a masked word in a given context. Human language communication is …