New Metric pBA Corrects Performance Estimation Bias in Imbalanced Classification
A recent study published on arXiv (2604.26024) presents a new metric called predicted-weighted balanced accuracy (pBA), designed to address performance estimation bias in imbalanced classification involving minority subconcepts. Traditional class-level evaluations often overlook differences among subconcepts, and standard metrics tend to favor larger minority subconcepts. Although utility-based reweighting can reduce bias by utilizing true subconcept labels, these labels are seldom accessible during testing. The innovative pBA substitutes missing subconcept labels with predicted posterior probabilities derived from a multiclass subconcept model, establishing evaluation weights based on the expected utility of this posterior. Tests conducted on tabular benchmarks, medical imaging, and text datasets reveal that unweighted scores may be deceptive in situations of within-class imbalance.
Key facts
- arXiv:2604.26024v1
- Announce Type: cross
- Class-level evaluation conceals performance disparities across subconcepts
- Common evaluation measures for imbalanced classification are biased toward larger minority subconcepts
- Utility-based reweighting using true subconcept labels can mitigate bias
- True subconcept labels are rarely available at test time
- Predicted-weighted balanced accuracy (pBA) replaces subconcept labels with predicted posterior probabilities
- Experiments on tabular, medical-imaging, and text datasets
Entities
Institutions
- arXiv