Ayan Sengupta

Manifold-Preserving Transformers are Effective for Short-Long Range Encoding

2023 · arXiv (Cornell University)

Multi-head self-attention-based Transformers have shown promise in different learning tasks. Albeit these models exhibit significant improvement in understanding short-term and long-term contexts from sequences, encoders of Transformers and their variants fail to preserve layer-wise contextual …