REAP: Automated Curation of Coding Agent Benchmarks from Production Usage
This paper introduces REAP (Relevance and Execution-Audited Pipeline), an automated pipeline for curating benchmarks for AI coding agents directly from real developer-agent sessions without manual labeling. Existing evaluation methods like online A/B testing are slow and risky, shadow deployment lacks reproducibility, and public benchmarks do not match production workloads in language distribution, prompt style, or codebase structure. REAP addresses challenges such as untestable prompts, misaligned tests, and test flakiness that compromise reliability. Manual auditing is infeasible in monorepo settings, so REAP automates the curation process to produce in-distribution benchmarks from production usage.
Key facts
- REAP stands for Relevance and Execution-Audited Pipeline.
- It automates curation of benchmarks from real developer-agent sessions.
- No manual labeling is required.
- Existing methods: online A/B testing (weeks, user risk), shadow deployment (non-reproducible), public benchmarks (out-of-distribution).
- Challenges: untestable prompts, misaligned tests, test flakiness.
- Manual auditing infeasible in monorepo settings.
- REAP produces production-derived benchmarks.
- The paper is from arXiv:2604.01527v4.
Entities
Institutions
- arXiv