Ryan Whetten

An Analysis of Linear Complexity Attention Substitutes with BEST-RQ

2024 · arXiv (Cornell University)

Self-Supervised Learning (SSL) has proven to be effective in various domains, including speech processing. However, SSL is computationally and memory expensive. This is in part due the quadratic complexity of multi-head self-attention (MHSA). Alternatives for …