Conformal Prediction Benchmark for Imbalanced High-Stakes Decision Support
A recent study released on arXiv (2607.27143) evaluates conformal prediction techniques in critical decision-making areas such as credit scoring, fraud detection, healthcare, and industrial safety. The findings reveal that conventional marginal conformal prediction significantly fails to cover rare minority classes, with coverage plummeting to as low as 0.5% in certain datasets. This benchmark assesses marginal CP, class-conditional (Mondrian) CP, and cost-controlled abstention strategies across 15 real-world imbalanced tabular datasets, utilizing 7 classification models, 3 probability calibration methods, and 10 random seeds, culminating in 3,150 experimental trials. The results indicate that Mondrian CP enhances minority-class coverage, yielding an average improvement of 61.7 percentage points.
Key facts
- Study benchmarks conformal prediction for imbalanced high-stakes decision support.
- Standard marginal CP under-covers minority classes, dropping to 0.5% coverage.
- Comparison includes marginal CP, Mondrian CP, and cost-controlled abstention.
- Uses 15 real-world imbalanced tabular datasets.
- Employs 7 classification models and 3 calibration techniques.
- Total of 3,150 experimental runs across 10 random seeds.
- Mondrian CP improves minority coverage by 61.7 percentage points on average.
- Published on arXiv with ID 2607.27143.
Entities
Institutions
- arXiv