Xiaodong Han

The Devil in Linear Transformer

2022 · arXiv (Cornell University)

Linear transformers aim to reduce the quadratic space-time complexity of vanilla transformers. However, they usually suffer from degraded performances on various tasks and corpus. In this paper, we examine existing kernel-based linear transformers and identify …