Daquan Zhou

ConvBERT: Improving BERT with Span-based Dynamic Convolution

2020 · arXiv (Cornell University)

Pre-trained language models like BERT and its variants have recently achieved impressive performance in various natural language understanding tasks. However, BERT heavily relies on the global self-attention block and thus suffers large memory footprint and …