Peter Liu

Unsupervised Pretraining for Sequence to Sequence Learning

2017

This work presents a general unsupervised learning method to improve the accuracy of sequence to sequence (seq2seq) models. In our method, the weights of the encoder and decoder of a seq2seq model are initialized with …

PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization

2020 · International Conference on Machine Learning

Recent work pre-training Transformers with self-supervised objectives on large text corpora has shown great success when fine-tuned on downstream NLP tasks including text summarization. However, pre-training objectives tailored for abstractive text summarization have not been …