AgentSearchBench: Benchmark for AI Agent Search in the Wild
AgentSearchBench has been launched by researchers as a comprehensive benchmark aimed at locating AI agents in practical situations. This benchmark is derived from almost 10,000 actual agents from various providers and frames agent search as both retrieval and reranking challenges, applicable to executable task queries and overarching task descriptions. It measures relevance through performance signals based on execution, tackling the difficulty of evaluating agent capabilities, which are frequently compositional and reliant on execution, using only textual descriptions. While experiments show consistent outcomes, the abstract does not detail specific results.
Key facts
- AgentSearchBench is a large-scale benchmark for agent search in the wild.
- Built from nearly 10,000 real-world agents across multiple providers.
- Formalizes agent search as retrieval and reranking problems.
- Evaluates relevance using execution-grounded performance signals.
- Addresses the challenge of assessing compositional and execution-dependent agent capabilities.
- Includes both executable task queries and high-level task descriptions.
- Existing benchmarks assume well-specified functionalities or controlled candidate pools.
- The benchmark aims to study realistic agent search scenarios.
Entities
Institutions
- arXiv