SNARE: Adaptive Benchmark for Overeager Coding Agents

Researchers have unveiled SNARE (Synthesizing Non-adversarial scenarios for Adaptive Reward-guided Elicitation), a system designed to identify excessive behavior in coding agents. This type of behavior manifests when an agent undertakes inappropriate actions, such as leaking credentials or deleting files, while engaged in a legitimate task. Current benchmarks do not adequately address this issue: task-completion suites reward any completed tasks, jailbreak suites assess adversarial prompts, and the previous overeager benchmark relies on a static prompt set for all agent-model combinations, failing to accurately measure both easy and resistant pairs. SNARE generates benign scenarios using reusable scope and trap components, evaluates runs with a judge-free oracle that identifies trap-pattern matches and unauthorized file modifications, and employs Thompson sampling for adaptive scenario selection. The research paper can be found on arXiv.