Statistical Framework Quantifies AI Agent Reliability Under Perturbations

A new paper on arXiv (2605.10516) establishes a rigorous measurement science for AI agent reliability, introducing statistical methods to quantify consistency under semantically preserving perturbations. The framework uses U-statistics for output-level reliability and kernel-based metrics for trajectory-level stability, distinguishing between core capability and execution robustness. Experiments on three agentic benchmarks show that trajectory-level metrics offer greater diagnostic sensitivity than traditional pass@1 rates, revealing that minor task variations can cause complete strategy breakdowns even when agents possess requisite knowledge. The work provides mathematical tools to isolate where and why agents deviate.