Payal Bajaj

METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals

2022 · arXiv (Cornell University)

We present an efficient method of pretraining large-scale autoencoding language models using training signals generated by an auxiliary model. Originated in ELECTRA, this training strategy has demonstrated sample-efficiency to pretrain models at the scale of …