Echo-State Network Detects LLM Agent Failures in Real Time
A recent study posted on arXiv (2608.02464) introduces a new method for detecting failures in large language model (LLM) agents. This approach uses observable telemetry data instead of relying on a pricey second LLM for step evaluation. It employs a one-class echo-state-network ensemble with CUSUM alarms, trained only on successful executions, measuring costs in microseconds. The technique was evaluated on 2,823 agent episodes across three frameworks, including local models like qwen2.5 and llama3.1, as well as a commercial API, gemini-2.5-flash. It accurately identified 71% of failures with a 5% false-alarm rate and showed improved performance over time, addressing common issues like loops and goal drift while being cost-effective.
Key facts
- The method uses a one-class echo-state-network ensemble with CUSUM alarms.
- Monitors cost microseconds per step and are trained only on healthy runs.
- Evaluation on 2,823 episodes across three frameworks and multiple models.
- Detects 0.71 of failures at a 5% false-alarm budget (AUROC 0.872).
- Advantage over memoryless baseline: +0.09 at ≤3 steps, +0.40 at ≥9 steps.
- Ranking transfers without retraining to AFTraj-2K (0.745) and ATBench (0.779).
- Addresses failures like loops, tool errors, goal drift, fabrication, and corrupted content.
- Paper available on arXiv with ID 2608.02464.
Entities
Institutions
- arXiv