John Hewitt

Designing and Interpreting Probes with Control Tasks

2019 · arXiv (Cornell University)

Probes, supervised models trained to predict properties (like parts-of-speech) from representations (like ELMo), have achieved high accuracy on a range of linguistic tasks. But does this mean that the representations encode linguistic structure or just …

XNMT: The eXtensible Neural Machine Translation Toolkit

2018 · arXiv (Cornell University)

This paper describes XNMT, the eXtensible Neural Machine Translation toolkit. XNMT distin- guishes itself from other open-source NMT toolkits by its focus on modular code design, with the purpose of enabling fast iteration in research …

Emergent linguistic structure in artificial neural networks trained by self-supervision

2020 · Proceedings of the National Academy of Sciences

This paper explores the knowledge of linguistic structure learned by large artificial neural networks, trained via self-supervision, whereby the model simply tries to predict a masked word in a given context. Human language communication is …

Lost in the Middle: How Language Models Use Long Contexts

2024 · Transactions of the Association for Computational Linguistics

Abstract While recent language models have the ability to take long contexts as input, relatively little is known about how well they use longer context. We analyze the performance of language models on two tasks …