Two-stage self-supervised training vision transformers for small datasets

After a series of achievements in the field of natural language processing, Transformers have shown promising results upon their introduction to computer vision, particularly under conditions of large-scale data. However, when faced with insufficient data, the performance of Vision Transformers(ViT) often falls short compared to Convolutional Neural Networks (CNNs), which are capable of capturing intrinsic biases in the data. In this paper, to address the performance disadvantage of ViT on small datasets, we propose a two-stage self-supervised training strategy. We enhance the ViT model by introducing Sequential Overlapping Patch Embedding (SOPE) and Improved Dynamic Aggregation Feed Forward (IDAFF) modules. Applying our approach to both single-block and multi-block ViT models on five commonly used small datasets (CIFAR10, CIFAR100, CINIC10, SVHN, Tiny-ImageNet) shows the efficacy of our approach. It narrows the performance difference between ViT and CNNs during training on small datasets from the ground up, and in some cases, even achieves better classification performance than CNNs. Our codes are available at: https://github.com/newer7/vosd.