Agent Evaluation Finality and Cross-Unit Separation

A recent study published on arXiv (2608.14940) investigates when agent evaluations can be deemed conclusive. The researchers contend that while existing evaluations assess models based on the final state of a halted run, labeling this as a definitive result necessitates two distinct criteria: outcome finality and cross-unit separation. Outcome finality guarantees that any delayed results are addressed, whereas cross-unit separation avoids any influence from previous runs. The paper outlines a completion argument detailing the evidence required for each determination, asserting that a final designation is warranted only when all factors that might alter the stated outcome are either resolved, confined, or acknowledged as uncertain. Through a controlled replay, the authors reveal that endpoint and terminal labels vary for each delayed operation, underscoring the importance of meticulous evaluation design.