Agentic AI Evaluation Scores Found Systematically Unreliable in New Study

A recent empirical study published on arXiv (paper 2608.00794) indicates that the automated benchmarks employed to assess agentic AI systems are considerably less reliable than previously recognized. The research highlights three interconnected layers of reliability issues. Initially, tasks are increasingly produced by language models; audits of ten widely-used benchmarks revealed validity issues in seven and reporting deficiencies in all ten. Next, human users are being substituted with LLM simulators, yet calibration studies show inter-simulator discrepancies of up to 9 percentage points and consistent miscalibration, especially for speakers of non-Standard American English. Lastly, a structured review of 55 papers indicates that around 82% utilize inter-rater reliability (IRR) metrics that are either mismatched, incomplete, or missing. These shortcomings multiply rather than simply add, resulting in a significantly lower overall reliability of evaluation scores than individual metrics imply. The implications of these findings are critical for deployment choices, safety certifications, and regulatory compliance that depend on these evaluations. The paper was introduced as a new submission on arXiv, with the abstract outlining its methodology and findings, emphasizing the urgent need for more stringent evaluation practices in the fast-evolving domain of agentic AI.