AI Evaluation Should Measure Verification Cost, Not Correctness Alone

A recent study published on arXiv (2608.08709) contends that existing metrics for evaluating AI generative models are inadequate, primarily because they prioritize the accuracy of outputs while neglecting the verification costs associated with those outputs. The researchers introduce the term Verification-Cost Errors (VCEs), which refer to incorrect input-output pairs that a certain percentage of verifiers cannot detect within a specified verification budget. Unlike typical hallucinations, VCEs are characterized by the inability to correctly identify outputs within budget constraints rather than by the attributes of the outputs themselves. The paper suggests that verification costs in relation to deployment budgets represent an operational aspect often overlooked in current evaluations, indicating a need to reassess how AI models are judged, focusing on practical verification challenges rather than solely on correctness.