Neural Stochastic Differential Equations.

Deep neural networks whose parameters are distributed according to typical initialization schemes exhibit undesirable properties that can emerge as the number of layers increases. These issues include a vanishing dependency on the input and a concentration on restrictive families of functions including constant functions. We address these problems by considering the limit of infinite total depth and examine the conditions under which we achieve convergence to well-behaved continuous-time processes. Doing so we establish the connection between infinitely deep residual networks and solutions to stochastic differential equations, i.e. diffusion processes. We show that deep neural networks satisfying such connection don't suffer from the mentioned pathologies and analyze the SDE limits to shed light on their behavior.