Shenghao Yang

Positional Attention: Expressivity and Learnability of Algorithmic Computation

2024 · arXiv (Cornell University)

There is a growing interest in the ability of neural networks to execute algorithmic tasks (e.g., arithmetic, summary statistics, and sorting). The goal of this work is to better understand the role of attention in …