Machine Learning for Reducing the Effort of Conducting Systematic Reviews in SE

Objective : To investigate whether machine learning and text-based data\nmining can be used to support the primary studies selection process and decrease the needed efforts\nin systematic reviews conducted in the context of SE.\nResearch Design : A test collection was built from 3 systematic reviews used in previous work in\nthe context of SE. The proposed probabilistic classifier based on Bayes’ Theorem was constructed to\npredict and classify each article as containing high-quality evidence to warrant inclusion in study\nselection process or not. Feature engineering techniques were applied to the abstract-based features.\nCross-validation experiments were performed to evaluate the efficiency of the document classifier.\nThree metrics - precision, recall and specificity were used together to measure the classification\nperformance. We assume that a recall rate of 0.9 or higher is required for the classifier to identify an\nsufficient quantity of relevant papers. As long as recall is at least 0.9, the Precision and Specificity\nshould be as high as possible,.\nResults : From the hold-out cross validation experiment, the precision achieved with the classifier\nfor two systematic review topics, was 93%, while 79% for another systematic review topic. The\nresults of leave-one-out cross validation experiment were presented in three Confusion Matrix,\nwhich in detail indicated that the precision achieved with the classifier for the three systematic\nreview topics was promising in terms of predicting relevant abstracts while relatively poor in terms\nof excluding irrelevant articles.\nConclusion : The classifier based on Bayes’ Theorem has strong potential for performing the\nsystematic review classification tasks in software engineering. The approach presented in this paper\ncould be considered as a possible technique for assisting labor-intensive primary studies’ selection\nprocess in an SLR.