Chinese abstractive summarization based on pre-trained module assisted with noise co-attention

The encoder-decoder framework underpins the majority of Chinese abstract generative models. Although the generated summaries are comparable to the source text, they invariably suffer from generating inaccurate or redundant words, which can result in imprecise content or insufficient summaries. To address this issue, we propose a Roberta-based model with noise co-attention. In our way, we propose a novel noise co-attention to correct and remove the learned biased and redundant information from the source text. We first use the pre-trained model ROBERTA to obtain high-quality word vector representations and then build UNILM based on the first step, in particular adding noise co-attention layer to guide the summary generation process. Related experiments are being carried out on the large Chinese abstract dataset LCSTS, using ROUGE as the evaluation metric. According to the results, our invention surpasses numerous cutting-edge models for the summary generation and offers greater improvements over baseline methods.