Sainbayar Sukhbaatar

End-To-End Memory Networks

2015 · arXiv (Cornell University)

We introduce a neural network with a recurrent attention model over a possibly large external memory. The architecture is a form of Memory Network (Weston et al., 2015) but unlike the model in that work, …

Adaptive Attention Span in Transformers

2019

We propose a novel self-attention mechanism that can learn its optimal attention span. This allows us to extend significantly the maximum context size used in Transformer, while maintaining control over their memory footprint and computational …