Target Permutation for Feature Significance and Applications in Neural Networks

Statistical techniques like generalized linear models have always been the indispensable tools for understanding relationships in data. However, over the last two decades, growth in data size and complexity has fueled the rise of powerful “black box” machine learning models (i.e., Deep Learning). Such models lack transparent mechanisms to evaluate the importance and contributions of individual features. This may be unacceptable ethically and/or legally. Moreover, the inability to precisely assess contributions from features complicates model validation and understanding. In this paper, we propose a permutation test for feature importance in differentiable models with a focus on neural networks. Unlike existing permutation-based methods, ours shuffles the target instead of the inputs. This change allows for simultaneous testing of all features and does not require the assumption of independence among inputs that is prevalent in current work in this area. Through extensive experiments, we empirically demonstrate that this permutation test can reveal highly nonlinear associations, is robust to multicollinearity among features, and can be used to filter unnecessary inputs, while preserving or improving models' predictive performance in both classification and regression.