Implementation Lottery: Flawed Evaluation in Automated Research

A recent study published on arXiv (2607.26587) uncovers a critical issue in the evaluation of automated research systems, termed the "implementation lottery." These systems rely on experimental scores to determine which concepts to keep; however, a single test only assesses one version of an idea. This approach leads to a structural discrepancy, as the conclusions drawn about an idea hinge on the specific implementation chosen. The authors introduce the Idea Reliability Audit, which assesses idea reliability by validating and freezing candidate cards, sampling new-session implementations, employing outcome-blind fidelity labels, and rerunning preserved artifacts. It reports on idea ICC and leave-one-implementation-out (LOO) winner reversal. In contrast to previous studies that focus on task repetition, this research emphasizes idea repetition. Across 312 assignments involving 13 tabular tasks and two coding-agent configurations, significant implementation variance highlights the extent of the lottery's impact.