Theory of Neural Language Models (Dagstuhl Seminar 25282)

This report documents the program and the outcomes of Dagstuhl Seminar 25282 "Theory of Neural Language Models". The seminar aimed to bring researchers together to lay a foundation for continued work on the theory of neural language models, focusing on questions including: How do transformers, RNNs, other NLMs, and their variants, compare with one another in expressivity and trainability? How do the successes and failures of NLMs predicted by theoretical models manifest in practice? What modifications, or what wholly new architectures, are suggested by the theory?