David Dohan

Evolving modular neural sequence architectures with genetic programming

2018 · Proceedings of the Genetic and Evolutionary Computation Conference Companion

Automated architecture search has demonstrated significant success for image data, where reinforcement learning and evolution approaches now outperform the best human designed networks ([12], [8]). These successes have not transferred over to models dealing with …

Rethinking Attention with Performers

2020 · arXiv (Cornell University)

We introduce Performers, Transformer architectures which can estimate regular (softmax) full-rank-attention Transformers with provable accuracy, but using only linear (as opposed to quadratic) space and time complexity, without relying on any priors such as sparsity …

QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension

2018 · arXiv (Cornell University)

Current end-to-end machine reading and question answering (Q\&A) models are primarily based on recurrent neural networks (RNNs) with attention. Despite their success, these models are often slow for both training and inference due to the …