Dieuwke Hupkes

Masked Language Modeling and the Distributional Hypothesis: Order Word\n Matters Pre-training for Little

2021 · arXiv (Cornell University)

A possible explanation for the impressive performance of masked language\nmodel (MLM) pre-training is that such models have learned to represent the\nsyntactic structures prevalent in classical NLP pipelines. In this paper, we\npropose a different explanation: MLMs …

The ICL Consistency Test

2023 · arXiv (Cornell University)

Just like the previous generation of task-tuned models, large language models (LLMs) that are adapted to tasks via prompt-based methods like in-context-learning (ICL) perform well in some setups but not in others. This lack of …

The emergence of number and syntax units in

2019

Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, Marco Baroni. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and …

Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little

2021 · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

A possible explanation for the impressive performance of masked language model (MLM) pre-training is that such models have learned to represent the syntactic structures prevalent in classical NLP pipelines. In this paper, we propose a …