AI Security Benchmarks Face Critical Flaws: New Research Identifies Three Core Challenges

A new paper from arXiv (submitted May 2025) identifies three fundamental weaknesses in benchmarks used to evaluate AI agents for security-critical roles. The authors characterize benchmark vulnerabilities, temporal staleness, and runtime uncertainty as core challenges that undermine current security evaluations. They outline practical directions for building more robust and trustworthy evaluation frameworks.