Threshold Tuning and Resampling for Imbalanced Classification: A 45-Task Study

A recent study released on arXiv (2608.16147) questions the prevalent method of assessing class-imbalance strategies using only one benchmark dataset. The authors reveal that insights gained from such limited evaluations can be deceptive. Analyzing the publicly available Kaggle credit-card fraud dataset with a leakage-free nested cross-validation method, they discovered that a standard Random Forest model at the default threshold of 0.5 yields an F1 score of 0.861 ± 0.021, with threshold adjustments offering no improvement (delta-F1 = -0.002). This indicates that for well-calibrated ensembles, managing imbalance may not be necessary. However, applying the same method to 45 binary tasks with imbalance ratios from 1:1.5 to 1:178 (totaling 2,025 model fits across four model families) led to a different conclusion. Random Forest showed the greatest advantage from threshold tuning (delta-F1 = +0.101 ± 0.134), while the other three model families exhibited varying results. The findings highlight the dangers of depending on single-dataset evaluations and call for broader benchmarking in imbalanced classification studies.