Understanding the Role of Adversarial Regularization in Supervised\n Learning

Despite numerous attempts sought to provide empirical evidence of adversarial\nregularization outperforming sole supervision, the theoretical understanding of\nsuch phenomena remains elusive. In this study, we aim to resolve whether\nadversarial regularization indeed performs better than sole supervision at a\nfundamental level. To bring this insight into fruition, we study vanishing\ngradient issue, asymptotic iteration complexity, gradient flow and provable\nconvergence in the context of sole supervision and adversarial regularization.\nThe key ingredient is a theoretical justification supported by empirical\nevidence of adversarial acceleration in gradient descent. In addition,\nmotivated by a recently introduced unit-wise capacity based generalization\nbound, we analyze the generalization error in adversarial framework. Guided by\nour observation, we cast doubts on the ability of this measure to explain\ngeneralization. We therefore leave as open questions to explore new measures\nthat can explain generalization behavior in adversarial learning. Furthermore,\nwe observe an intriguing phenomenon in the neural embedded vector space while\ncontrasting adversarial learning with sole supervision.\n