BulkPR-Bench: Benchmarking Queue-Level Governance of Interacting Pull Requests
The introduction of BulkPR-Bench marks a new standard for assessing coding agents' proficiency in handling a queue of interacting pull requests (PRs). This benchmark, outlined in arXiv paper 2608.02685, fills a void in current coding-agent evaluations, which often concentrate on isolated outcomes or predetermined change sequences. BulkPR-Bench challenges agents to identify significant PR relationships and produce a large executable subset in a safe order, adhering to a rolling-release protocol. It includes 581 newly created candidate PRs from 18 actual repositories' frozen snapshots. The benchmark verifies the gold relation graph through state-by-state repository execution, incorporating hidden safety checks, and employs an exact oracle for determining the largest safe subset. The key metric, Relational Delivery Score (RDS), assesses safe delivery and accurate rejection across relation graphs, contributing significantly to the governance of intricate, interactive development processes.
Key facts
- BulkPR-Bench is a new benchmark for queue-level governance of interacting pull requests.
- It is described in arXiv paper 2608.02685.
- The benchmark includes 581 newly authored candidate PRs.
- These PRs are on frozen snapshots of 18 real repositories.
- It uses a rolling-release protocol.
- The primary metric is Relational Delivery Score (RDS).
- The benchmark validates the gold relation graph via state-by-state repository execution.
- An exact oracle computes the largest safe subset.
Entities
Institutions
- arXiv