Zhengyan Zhang

SHUOWEN-JIEZI: Linguistically Informed Tokenizers For Chinese Language Model Pretraining

2021 · arXiv (Cornell University)

Conventional tokenization methods for Chinese pretrained language models (PLMs) treat each character as an indivisible token (Devlin et al., 2019), which ignores the characteristics of the Chinese writing system. In this work, we comprehensively study …

Plug-and-Play Knowledge Injection for Pre-trained Language Models

2023 · arXiv (Cornell University)

Injecting external knowledge can improve the performance of pre-trained language models (PLMs) on various downstream NLP tasks. However, massive retraining is required to deploy new knowledge injection methods or knowledge bases for downstream tasks. In …

ERNIE: Enhanced Language Representation with Informative Entities

2019

Neural language representation models such as BERT pre-trained on large-scale corpora can well capture rich semantic patterns from plain text, and be fine-tuned to consistently improve the performance of various NLP tasks. However, the existing …

KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation

2019 · arXiv (Cornell University)

Pre-trained language representation models (PLMs) cannot well capture factual knowledge from text. In contrast, knowledge embedding (KE) methods can effectively represent the relational facts in knowledge graphs (KGs) with informative entity embeddings, but conventional KE …

KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation

2021 · Transactions of the Association for Computational Linguistics

Abstract Pre-trained language representation models (PLMs) cannot well capture factual knowledge from text. In contrast, knowledge embedding (KE) methods can effectively represent the relational facts in knowledge graphs (KGs) with informative entity embeddings, but conventional …