An Interpretable Regularization Method Based on Minimizing Mutual Information

Deep learning model is popular and widely used. Its generalization performance is superior to traditional machine learning methods. Traditional statistical learning theory cannot explain its success. Most of network architectures are designed artificially and empirically. There are few methods to design new structures and mathematical tools to evaluate feature representation capabilities of networks. To address this issue, based on information theory, we propose an interpretable regularization method named Minimize Mutual Information Method (MMIM), which reduce the generalization error by minimizing the mutual information between hidden layer neurons. We derive a tighter generalization error boundary and propose a two-step method to reduce the neural network generalization error boundary. The experimental results verify the effectiveness of our proposed MMIM on Fashion-MNIST dataset and CIFAR datasets. Inspired by ensemble learning, we discuss why our method is effective and suggest how to design new networks.