Kharlamov, Evgeny

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

2019 · arXiv (Cornell University)

As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging. In this work, we …