Fishr: Invariant Gradient Variances for Out-of-Distribution\n Generalization

Learning robust models that generalize well under changes in the data\ndistribution is critical for real-world applications. To this end, there has\nbeen a growing surge of interest to learn simultaneously from multiple training\ndomains - while enforcing different types of invariance across those domains.\nYet, all existing approaches fail to show systematic benefits under controlled\nevaluation protocols. In this paper, we introduce a new regularization - named\nFishr - that enforces domain invariance in the space of the gradients of the\nloss: specifically, the domain-level variances of gradients are matched across\ntraining domains. Our approach is based on the close relations between the\ngradient covariance, the Fisher Information and the Hessian of the loss: in\nparticular, we show that Fishr eventually aligns the domain-level loss\nlandscapes locally around the final weights. Extensive experiments demonstrate\nthe effectiveness of Fishr for out-of-distribution generalization. Notably,\nFishr improves the state of the art on the DomainBed benchmark and performs\nconsistently better than Empirical Risk Minimization. Our code is available at\nhttps://github.com/alexrame/fishr.\n