Projectibility in AI Evaluation: When Benchmark Inferences Fail to Compose

A recent paper published on arXiv (2607.26159) highlights a core epistemic challenge in evaluating AI: while individual inferences from benchmark outcomes may be justified, the overall reasoning may not be reliable. The authors contend that validity-focused methods necessitate proof for each assertion, but complications arise when the focus of one study differs from the next or when changes occur in the system, population, outcome, or conditions. Additionally, shared data or model lineage can create dependencies between seemingly independent supports. The paper, referencing Goodman's problem of rival extensions and argument-based validity, presents the idea of "projectibility," which questions whether a limited extension from observed to unobserved cases is justified. The key assertion is that the non-compositional nature of inferences compromises the validity of numerous AI benchmark assessments.