Liu, Qun · PaperMetrix

Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order

2020 · arXiv (Cornell University)

Masked language model and autoregressive language model are two types of language models. While pretrained masked language models such as BERT overwhelm the line of natural language understanding (NLU) tasks, autoregressive language models such as …

PanGu-Bot: Efficient Generative Dialogue Pre-training from Pre-trained Language Model

2022 · arXiv (Cornell University)

In this paper, we introduce PanGu-Bot, a Chinese pre-trained open-domain dialogue generation model based on a large pre-trained language model (PLM) PANGU-alpha (Zeng et al.,2021). Different from other pre-trained dialogue models trained over a massive …

Pre-training Language Models with Deterministic Factual Knowledge

2022 · arXiv (Cornell University)

Previous works show that Pre-trained Language Models (PLMs) can capture factual knowledge. However, some analyses reveal that PLMs fail to perform it robustly, e.g., being sensitive to the changes of prompts when extracting factual knowledge. …

Learning to Edit: Aligning LLMs with Knowledge Editing

2024 · arXiv (Cornell University)

Knowledge editing techniques, aiming to efficiently modify a minor proportion of knowledge in large language models (LLMs) without negatively impacting performance across other inputs, have garnered widespread attention. However, existing methods predominantly rely on memorizing …

TinyBERT: Distilling BERT for Natural Language Understanding

2019 · arXiv (Cornell University)

Language model pre-training, such as BERT, has significantly improved the performances of many natural language processing tasks. However, pre-trained language models are usually computationally expensive, so it is difficult to efficiently execute them on resource-restricted …

PanGu-$α$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation

2021 · arXiv (Cornell University)