Neural Rule Ensembles: Encoding Sparse Feature Interactions into Neural\n Networks

Artificial Neural Networks form the basis of very powerful learning methods.\nIt has been observed that a naive application of fully connected neural\nnetworks to data with many irrelevant variables often leads to overfitting. In\nan attempt to circumvent this issue, a prior knowledge pertaining to what\nfeatures are relevant and their possible feature interactions can be encoded\ninto these networks. In this work, we use decision trees to capture such\nrelevant features and their interactions and define a mapping to encode\nextracted relationships into a neural network. This addresses the\ninitialization related concern of fully connected neural networks. At the same\ntime through feature selection it enables learning of compact representations\ncompared to state of the art tree-based approaches. Empirical evaluations and\nsimulation studies show the superiority of such an approach over fully\nconnected neural networks and tree-based approaches\n