Anticipating Spectrogram Classification Error With Combinatorial Coverage Metrics

Recently, combinatorial interaction testing (CIT) has been applied to machine learning. Recent results demonstrate that combinatorial coverage metrics can correlate with classification error. However, these methods have not been applied to the area of cognitive communications. In this paper, we show that combinatorial coverage metrics can be used to anticipate spectrogram classification error for convolutional neural networks (CNNs) for batch sizes of 100-500 spectrograms. We propose a method that combines deep autoencoders, metric learning, and k-means clustering to learn a low-dimensional, discrete representation of spectrograms for coverage analysis. Our results show strong correlations which suggest that combinatorial coverage metrics can be used to anticipate spectrogram classification error. This finding identifies coverage as a new class of solution methods for monitoring and testing cognitive radios.