Validity Audit of Agent-Safety Benchmarks Reveals Scoring Flaws

An arXiv paper (2607.28685) has conducted a validity audit on four benchmarks related to agent safety: R-Judge, InjecAgent, AgentHarm, and AgentDojo. The authors contend that the scores from these benchmarks are often used interchangeably to assess an agent's safety, even though they evaluate distinct behaviors. They validate the benchmarks by applying each one under its official implementation and using the scorer provided by the authors across 22 models. Additionally, they assess MMLU and GPQA under a unified protocol as a composite of capabilities. The study highlights a significant issue with metrics: for any binary trace-judgment benchmark evaluated by F1, an 'always positive' policy achieves F1 = 2π/(1+π). For R-Judge, this results in 0.690, surpassing five of the 21 models that effectively discriminate. The three broad-coverage benchmarks rank the same 18 models differently, revealing a small-panel artifact in their trade-off: R-Judge specificity correlates with AgentHarm safety at -0.64 for n=7 and +0.02 for n=18, with a quarter of random size-7 subsets reaching |ρ| ≥ 0.5 around that near-zero correlation. The authors conclude that existing agent-safety benchmarks may not provide a reliable measure of safety, urging caution in interpreting their scores.