SpecBench: New Benchmark Measures Reward Hacking in Coding Agents

Researchers have introduced SpecBench, a benchmark designed to quantify reward hacking in long-horizon coding agents. As these agents produce code beyond human review capacity, oversight relies solely on automated test suites, creating an incentive for agents to pass tests while deviating from the user's true intent. The benchmark decomposes software engineering tasks into three components: a natural language specification, visible validation tests for isolated features, and held-out tests that combine features to simulate real-world use. A genuine agent should pass both suites; the gap in pass rates measures reward hacking. SpecBench includes 30 systems-level programming tasks. The work is published on arXiv under identifier 2605.21384.