Yiheng Huang

Masked Pre-trained Encoder base on Joint CTC-Transformer

2020 · arXiv (Cornell University)

This study (The work was accomplished during the internship in Tencent AI lab) addresses semi-supervised acoustic modeling, i.e. attaining high-level representations from unsupervised audio data and fine-tuning the parameters of pre-trained model with supervised data. …

Speech-XLNet: Unsupervised Acoustic Model Pretraining for Self-Attention Networks

2020

Self-attention network (SAN) can benefit significantly from the bi-directional representation learning through unsupervised pretraining paradigms such as BERT and XLNet.In this paper, we present an XLNet-like pretraining scheme "Speech-XLNet" to learn speech representations with self-attention …