Weiran Wang

TransformerFAM: Feedback attention is working memory

2024 · arXiv (Cornell University)

While Transformers have revolutionized deep learning, their quadratic attention complexity hinders their ability to process infinitely long inputs. We propose Feedback Attention Memory (FAM), a novel Transformer architecture that leverages a feedback loop to enable …