Shikhar Murty

Pushdown Layers: Encoding Recursive Structure in Transformer Language Models

2023 · arXiv (Cornell University)

Recursion is a prominent feature of human language, and fundamentally challenging for self-attention due to the lack of an explicit recursive-state tracking mechanism. Consequently, Transformer language models poorly capture long-tail recursive structure and exhibit sample-inefficient …

Hierarchical Losses and New Resources for Fine-grained Entity Typing and Linking

2018

Extraction from raw text to a knowledge base of entities and fine-grained types is often cast as prediction into a flat set of entity and type labels, neglecting the rich hierarchies over types and entities …