An Introduction of Deep Learning Based Word Representation Applied to Natural Language Processing

In the area of Natural Language Processing, high-quality word representations play key roles in neural language processing tasks. Recently, various model designs and methods have blossomed in the domain of word representation. In this paper, we will explain the theories of two major language models such as autoencoding (AE) and autoregressive (AR), and illustrate the architectures of several notable examples of AE and AR including ELMo, GPT, BERT and XLnet. By comparing the pros and cons of these models theoretically and the performances of these models in experiments, we will deepen the understanding of various learning methods and realize the trend in the development of language models.