Joshua Ainslie

Big Bird: Transformers for Longer Sequences

2020 · arXiv (Cornell University)

Transformers-based models, such as BERT, have been one of the most successful deep learning models for NLP. Unfortunately, one of their core limitations is the quadratic dependency (mainly in terms of memory) on the sequence …

LongT5: Efficient Text-To-Text Transformer for Long Sequences

2022 · Findings of the Association for Computational Linguistics: NAACL 2022

Recent work has shown that either (1) increasing the input length or (2) increasing model size can improve the performance of Transformer-based neural models. In this paper, we present LongT5, a new model that explores …