Why ROC-AUC Alone Is Insufficient for Highly Imbalanced Data: In-Depth Evaluation of MCC, F2-Score, H-Measure, and AUC-Based Metrics for Rare-Event Binary Classification

This study re-evaluates ROC-AUC for binary classification under severe class imbalance (<3% positives). Despite widespread use, ROC-AUC can mask operationally salient differences among classifiers when false-positive and false-negative costs are asymmetric. Using three benchmarks, credit-card fraud detection (0.17%), yeast protein localization (1.35%), and ozone level detection (2.9%), we compare ROC-AUC with Matthews Correlation Coefficient (MCC), F2-score, H-measure, and area under the precision–recall curve (PR-AUC). Our empirical analyses span 20 classifier–sampler configurations per dataset, four classifiers (Logistic Regression, Random Forest, XGBoost, and CatBoost) crossed with four oversampling methods plus a no-resampling baseline (no resampling, SMOTE, Borderline-SMOTE, SVM-SMOTE, ADASYN). ROC-AUC exhibits pronounced ceiling effects, yielding high scores even for underperforming pipelines. In contrast, MCC and F2 align more closely with deployment-relevant costs and achieve the highest Kendall’s τ rank concordance across datasets; PR-AUC provides threshold-independent ranking, and H-measure integrates cost sensitivity. We quantify uncertainty and differences using stratified bootstrap confidence intervals, DeLong’s test for ROC-AUC, and Friedman–Nemenyi critical-difference diagrams, which collectively underscore ROC-AUC’s limited discriminative value in rare-event settings. The findings support a shift to a multi-metric evaluation framework, recommending MCC and F2 as primary indicators, supplemented by PR-AUC and H-measure where ranking granularity and principled cost integration are required. This evidence encourages researchers and practitioners to move beyond sole reliance on ROC-AUC when evaluating classifiers in highly imbalanced data.