HG-CRC: Hierarchical Group-Conditional Conformal Risk Control for Selective Prediction in Language Models
The newly developed Hierarchical Group-Conditional Conformal Risk Control (HG-CRC) framework aims to rectify the shortcomings of traditional conformal risk control (CRC), which fails to deliver risk assurances for specific groups within large language models. While standard CRC guarantees overall population risk, it can breach subgroup limits with minor changes in group composition, showing failures in as many as 47% of cases. HG-CRC implements a Bonferroni correction across a user-defined hierarchy and adopts a leaf-first strategy to determine the most precise applicable threshold, reverting to broader nodes when necessary. This approach only requires a held-out calibration set without the need for retraining. The evaluation included three models, such as Qwen, and the findings are available on arXiv (2607.24562).
Key facts
- Standard CRC violates subgroup budgets in up to 47% of trials under mild group composition shift.
- HG-CRC enforces simultaneous risk guarantees across all nodes of a user-defined group hierarchy.
- HG-CRC uses a Bonferroni correction over nodes and a leaf-first policy.
- It requires only a held-out calibration set with no retraining.
- Evaluated on three models including Qwen.
- Published on arXiv with ID 2607.24562.
Entities
Institutions
- arXiv