Julián Salazar

Self-attention Networks for Connectionist Temporal Classification in Speech Recognition

2019

The success of self-attention in NLP has led to recent applications in end-to-end encoder-decoder architectures for speech recognition. Separately, connectionist temporal classification (CTC) has matured as an alignment-free, non-autoregressive approach to sequence transduction, either by …

Transformers without Tears: Improving the Normalization of Self-Attention

2019 · arXiv (Cornell University)

We evaluate three simple, normalization-centric changes to improve Transformer training. First, we show that pre-norm residual connections (PreNorm) and smaller initializations enable warmup-free, validation-based training with large learning rates. Second, we propose $\ell_2$ normalization with …