Selective Risk Control for Document Extraction: Three Failure Modes and a Validity Ladder

An arXiv paper (2608.14639) examines the effectiveness of per-field selective risk management in document-extraction systems, where a field is deemed acceptable only if the error rate among accepted fields is maintained at a predetermined alpha level. Analyzing 13,859 fields from 800 CORD receipts processed by claude-sonnet-5 (with a 49.0% accuracy), the research identifies three failure modes that can lead to unintentional breaches of the risk contract. Document clustering causes a design effect of 1.84-2.45, increasing variance. Score-refit leakage results in a coverage of 0.416 at a risk of 0.127, violating alpha=0.10 in 95% of cases. Additionally, a tie-mass pathology reduces coverage from 0.030 to 0.001. The authors suggest a validity ladder to categorize solutions based on their reliability. A fit/val split protocol is demonstrated to restore expected selective risk control for a learned fusion, achieving coverage of 0.318 at a risk of 0.096 with a nominal alpha of 0.10, while a production variant reaches 0.326 coverage. Nevertheless, this average point has seen risk surpassing alpha in 47.5% of resplits, highlighting that the guarantee is inconsistent. The paper offers a comprehensive diagnosis and a framework to aid practitioners in selecting suitable risk-control strategies according to their violation tolerance.