RepoProbe: New Benchmark for Repository-Scale Code Understanding with LLMs
RepoProbe, a novel benchmark, has been launched to assess the comprehension of large language models (LLMs) regarding complete software repositories. This benchmark is outlined in a paper available on arXiv (2608.04783) and addresses shortcomings in current evaluation techniques that depend on GitHub Issues bug reports, which often enable models to rely on pattern recognition rather than true understanding. Instead, RepoProbe utilizes open-ended inquiries from GitHub Discussions to evaluate architectural insight instead of merely identifying defects. Additionally, the paper introduces a Checklist-Based Verification Protocol aimed at enhancing the consistency of LLM-as-a-Judge scoring, which currently exhibits significant variability and low clarity. The research underscores the problem of 'Edit Bias,' where models suggest code changes without fully comprehending the existing structure. This benchmark seeks to redirect attention from function-level generation to repository-wide support, offering a more thorough evaluation of LLMs in software engineering scenarios.
Key facts
- RepoProbe is a new benchmark for evaluating repository-level code understanding.
- It uses open-ended Q&A from GitHub Discussions instead of bug reports.
- The benchmark addresses 'Edit Bias' in LLM code generation.
- A Checklist-Based Verification Protocol is proposed for evaluation.
- Existing benchmarks rely on GitHub Issues, which allow pattern matching.
- LLM-as-a-Judge scalar scoring has high variance and low interpretability.
- The paper is available on arXiv with ID 2608.04783.
- The focus is on architectural inquiries rather than defect reporting.
Entities
Institutions
- arXiv
- GitHub