Paul Hongsuck Seo

Learning Correlation Structures for Vision Transformers

2024 · arXiv (Cornell University)

We introduce a new attention mechanism, dubbed structural self-attention (StructSA), that leverages rich correlation patterns naturally emerging in key-query interactions of attention. StructSA generates attention maps by recognizing space-time structures of key-query correlations via convolution …