Agent Retrieval Bench: Benchmarking Repository Context Retrieval for Coding Agents
The preprint arXiv 2607.24882 presents Agent Retrieval Bench, a benchmark designed to assess the initial retrieval phase of coding agents, which involves identifying pertinent repository files prior to patch generation. This benchmark is derived from authentic coding workflow signals and utilizes static base-commit repositories, determining relevance based on the agent's immediate needs rather than semantic similarity. It features four positive-retrieval tasks: code2test, comment2context, trace2code, and edit2ripple, along with a fifth subset for selective retrieval that includes natural no-gold cases and counterfactual wrong-repository controls. The dataset consists of 427 samples from 25 repositories, including 345 positive instances, 50 natural no-gold cases, and 32 counterfactual controls, along with a corpus of 308 base-commits.
Key facts
- Agent Retrieval Bench is a file-level benchmark for coding agent context retrieval.
- Samples are built from real coding-workflow signals.
- Relevance is defined by what an agent needs next, not semantic similarity.
- Four positive-retrieval tasks: code2test, comment2context, trace2code, edit2ripple.
- A fifth subset evaluates selective retrieval with no-gold cases and counterfactual controls.
- Dataset contains 427 samples across 25 repositories.
- Includes 345 positive examples, 50 natural no-gold examples, 32 counterfactual controls.
- Corpus includes 308 base-commits.
Entities
Institutions
- arXiv