EcoAgent-Bench: New Benchmark for Budget-Conscious LLM Agents

A novel benchmark named EcoAgent-Bench has been launched to assess economic decision-making in large language model (LLM) agents while adhering to budget limitations. This benchmark is elaborated in a paper available on arXiv (2608.05519). Unlike conventional benchmarks that prioritize task completion, EcoAgent-Bench incorporates resource allocation as a core component of the task. Each task outlines priced actions and a defined budget, compelling agents to make strategic selections among local lookups, extensive searches, composite research tools, superior models, or human intervention. The benchmark features 304 tasks derived from real-world scenarios across five categories, adapted from GAIA, HotpotQA, and MuSiQue, focusing on four critical decisions: avoiding unnecessary escalation, escalating when local evidence is lacking, choosing a model tier, and halting on unsupported premises. The evaluation encompasses seven LLM agents in tool-API and workspace-CLI environments, along with four oracle scripted controls. Findings reveal that micro-averaged accuracy favors one-sided strategies, with always-escalate controls achieving high micro success yet struggling with save-oriented tasks. To tackle this issue, the authors suggest implementing an economic-consistency score, which reflects the lower accuracy on upgrades... (truncated for brevity)