Lin Tang

Batch Normalization Is Blind to the First and Second Derivatives of the Loss

2022 · arXiv (Cornell University)

In this paper, we prove the effects of the BN operation on the back-propagation of the first and second derivatives of the loss. When we do the Taylor series expansion of the loss function, we …