AGENTCHAOSBENCH: Benchmark for Detecting Runtime Faults in LLM Agentic Systems

A new benchmark called AGENTCHAOSBENCH has been launched to identify and locate runtime errors in LLM-based agentic systems using execution telemetry. It tackles the issue that reliability hinges on the entire execution process—including tool and model calls, guardrails, and inter-agent communications—rather than just the final output. Traditional evaluation methods that only look at the end results miss the reasons behind failures. This benchmark tests five diverse applications that use the Agent-to-Agent protocol and the Model Context Protocol for tool calls. It introduces ten different types of operational faults at various points and includes a no-fault control. The faults range from slow or unavailable tools to misrouted messages. The dataset generated includes 275 sanitized traces: 250 faulty runs covering ten fault types and 25 controls without faults. Each faulty trace is marked with its specific fault type and location, which helps in assessing fault detection and localization techniques. This research is detailed in a paper on arXiv with the identifier 2608.14680, which likely covers how the benchmark was created, the fault injection process, and its potential to enhance reliability in agentic systems.