Advanced similarity metrics for IP flow data analytics

Machine learning techniques provide powerful tools for analysis of encrypted network traffic. We present a novel set of features and a distance measure suitable for a wide variety of distance-based machine learning techniques useful in classification, clustering, and novelty detection in encrypted traffic flow data. The proposed features are given by distances between probability distributions of such characteristics as packet sizes or inter-arrival times. The proposed distance measure incorporates those features in a fractional l p -metric with p close to 0.1. The effectiveness of the distance measure is evaluated using an extensive labeled dataset containing 154 web services. The dataset was captured on the ISP backbone, and we present it as supplementary material. Application of k -nearest neighbors (kNN) algorithm in combination with the proposed distance measure and feature selection gives the classification accuracy of 91.0%. Comparison with a deep learning model shows that the kNN is competitive with state-of-the-art models. On a different publicly available dataset with a low number of classes, our approach reaches accuracy 99.6%, outperforming models presented in the literature. The benefits of novel features are further demonstrated using the LightGBM model, i.e. without relying on distance-based techniques. Besides direct classification, we have used the local outlier factor and rank-based detection algorithms to detect novel traffic flows. We show that their performance improves when using the proposed distance measure. Finally, to speed up traffic classification we compared the performance of kNN and Approximate Nearest Neighbors (ANN) algorithm, and applied the Affinity propagation clustering algorithm to select a suitable subset of kNN/ANN training points.