Distributed Bayesian Learning with Stochastic Natural-gradient\n Expectation Propagation and the Posterior Server

This paper makes two contributions to Bayesian machine learning algorithms.\nFirstly, we propose stochastic natural gradient expectation propagation (SNEP),\na novel alternative to expectation propagation (EP), a popular variational\ninference algorithm. SNEP is a black box variational algorithm, in that it does\nnot require any simplifying assumptions on the distribution of interest, beyond\nthe existence of some Monte Carlo sampler for estimating the moments of the EP\ntilted distributions. Further, as opposed to EP which has no guarantee of\nconvergence, SNEP can be shown to be convergent, even when using Monte Carlo\nmoment estimates. Secondly, we propose a novel architecture for distributed\nBayesian learning which we call the posterior server. The posterior server\nallows scalable and robust Bayesian learning in cases where a data set is\nstored in a distributed manner across a cluster, with each compute node\ncontaining a disjoint subset of data. An independent Monte Carlo sampler is run\non each compute node, with direct access only to the local data subset, but\nwhich targets an approximation to the global posterior distribution given all\ndata across the whole cluster. This is achieved by using a distributed\nasynchronous implementation of SNEP to pass messages across the cluster. We\ndemonstrate SNEP and the posterior server on distributed Bayesian learning of\nlogistic regression and neural networks.\n Keywords: Distributed Learning, Large Scale Learning, Deep Learning, Bayesian\nLearn- ing, Variational Inference, Expectation Propagation, Stochastic\nApproximation, Natural Gradient, Markov chain Monte Carlo, Parameter Server,\nPosterior Server.\n