E-Bench: Synthetic Benchmark for Multi-Step Tool-Use Agents

Researchers have introduced a new benchmark called E-Bench to evaluate large language models (LLMs) as agents that can perform multi-step tasks in changing settings. E-Bench consists of 323 tasks that create state changes across three different areas: Honor of Kings, QQ Music, and Tencent Meeting. By using graph-guided database filling, it effectively separates the creation of environments from task generation, allowing for reusable product environments without leftover components. The design features generator-solver asymmetry, presenting challenges that include both information and tool gaps, which require agents to uncover hidden data and make multiple tool calls before achieving a state change. This research aims to improve upon existing benchmarks that usually focus on isolated API calls or brief sequences. The full paper is accessible on arXiv under the reference 2607.23722.