Ziyang Wu

Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction

2024 · arXiv (Cornell University)

The attention operator is arguably the key distinguishing factor of transformer architectures, which have demonstrated state-of-the-art performance on a variety of tasks. However, transformer attention operators often impose a significant computational burden, with the computational …