Zhenzhong Lan

Contrastive Learning of Sentence Embeddings from Scratch

2023

Contrastive learning has been the dominant approach to train state-of-the-art sentence embeddings. Previous studies have typically learned sentence embeddings either through the use of human-annotated natural language inference (NLI) data or via large-scale unlabeled sentences …

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

2019 · arXiv (Cornell University)

Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations and longer training times. To …