AJ-Bench Introduces New Benchmark for AI Agent Evaluation in Complex Environments

A new benchmark called AJ-Bench has been introduced to systematically evaluate Agent-as-a-Judge approaches for verifying AI agent behaviors in complex environments. The benchmark addresses limitations of existing methods like rule-based verifiers and LLM-as-a-Judge models, which struggle with generalization beyond narrow domains. Agent-as-a-Judge works by actively interacting with environments and tools to gather verifiable evidence. AJ-Bench comprises 155 tasks and 516 annotated trajectories across three domains: search, data systems, and graphical user interfaces. It comprehensively assesses judge agents' capabilities in information acquisition, state verification, and process verification. Experiments show consistent performance improvements over LLM-as-a-Judge baselines while revealing substantial open challenges. The research was announced on arXiv with identifier 2604.18240v1. This work emerges as reinforcement learning continues to scale training of large language model-based agents, making reliable verification increasingly difficult.