Anirudh Ravula

Big Bird: Transformers for Longer Sequences

2020 · arXiv (Cornell University)

Transformers-based models, such as BERT, have been one of the most successful deep learning models for NLP. Unfortunately, one of their core limitations is the quadratic dependency (mainly in terms of memory) on the sequence …