OrchBench: Simulated Benchmark for Multi-Agent Orchestration Plans

Researchers have introduced OrchBench, a simulation-based benchmark designed to evaluate multi-agent orchestration plans in isolation. Complex tasks often require decomposition into parallel yet interdependent subtasks, making orchestration critical for multi-agent system performance. Traditional end-to-end evaluations conflate orchestration quality with worker capabilities, tool reliability, and environmental noise, and become expensive as workflow scale increases. OrchBench constructs directed acyclic graphs (DAGs) from real-world tasks, encoding task dependencies with controlled sizes and parallelism. Given a DAG, per-agent context limits, and an agent budget, the planner assigns subtasks to agents, specifies cross-agent information transfers, and retention ratios. A deterministic simulator then evaluates the plan without real execution. This approach isolates orchestration quality, reduces evaluation costs, and enables systematic benchmarking. The work is published on arXiv under identifier 2607.25656.