Pratyusha Sharma

Pushdown Layers: Encoding Recursive Structure in Transformer Language Models

2023 · arXiv (Cornell University)

Recursion is a prominent feature of human language, and fundamentally challenging for self-attention due to the lack of an explicit recursive-state tracking mechanism. Consequently, Transformer language models poorly capture long-tail recursive structure and exhibit sample-inefficient …