Nafis Sadeq

InforMask: Unsupervised Informative Masking for Language Model Pretraining

2022 · arXiv (Cornell University)

Masked language modeling is widely used for pretraining large language models for natural language understanding (NLU). However, random masking is suboptimal, allocating an equal masking rate for all tokens. In this paper, we propose InforMask, …