Sparse Meta Networks for Sequential Adaptation and its Application to\n Adaptive Language Modelling

Training a deep neural network requires a large amount of single-task data\nand involves a long time-consuming optimization phase. This is not scalable to\ncomplex, realistic environments with new unexpected changes. Humans can perform\nfast incremental learning on the fly and memory systems in the brain play a\ncritical role. We introduce Sparse Meta Networks -- a meta-learning approach to\nlearn online sequential adaptation algorithms for deep neural networks, by\nusing deep neural networks. We augment a deep neural network with a\nlayer-specific fast-weight memory. The fast-weights are generated sparsely at\neach time step and accumulated incrementally through time providing a useful\ninductive bias for online continual adaptation. We demonstrate strong\nperformance on a variety of sequential adaptation scenarios, from a simple\nonline reinforcement learning to a large scale adaptive language modelling.\n