arXiv Paper Proposes Corrected AI Coding Agent Benchmark Metric

A new study on arXiv (2608.14711) points out a major flaw in how the pass@k estimator, introduced by Chen et al. in 2021, is being used to assess AI coding agents. The authors argue that the parameter n is incorrectly interpreted; it should reflect the number of independent rollout attempts rather than the unit tests from a single submission. This misapplication leads to inflated performance metrics. They propose a new metric called reliability@k, where n indicates independent rollouts and c represents fully successful rollouts per (task, agent) pair. In tests, the flawed metric falsely boosts scores by 0.85-0.97. They also introduce security-adjusted reliability@k, focusing on rollouts that pass both functional and security tests. The complete study is available at https://arxiv.org/abs/2608.14711.