Revisiting Spatial Inductive Bias with MLP-Like Model

In recent years, deep convolution neural nets have produced outstanding results in vision tasks compared to previous methods. In the Transformer models with self-attention structure, the introduction of inductive bias of convolution operator has been studied. There is no doubt that the hard inductive bias of the convolution operator, i.e., the receptive field (locality) and spatial invariance (weight sharing) in the spatial dimension, is an essential factor in achieving high performance. We hypothesize that by placing additional constraints on the locality, we can obtain better inductive bias. In this paper, we propose an MLP-like model, content-aware token mixing MLP (CaMLP), in which locality and spatial invariance are tunable. We verify that smooth and outward-decaying locality plays an important role in token mixing of the model with relaxed spatial invariance in the CIFAR-10/ImageNet100 classification task.