ForestBench: Unified Graph Framework for Evaluating Multi-Agent Collaboration

A novel evaluation framework for multi-agent systems (MAS) leveraging large language models (LLMs) has been detailed in a paper on arXiv (2608.08605). This framework, called ForestBench, tackles the difficulty of assessing various MAS techniques by translating their execution traces into a common space of unified collaboration graphs. This enables the evaluation of different methods using the same representation, reference set, and metrics. Candidate graphs are assessed against a query-specific reference forest, which consists of verified-success graphs from the benchmark. Instead of prescribing a single best approach, the forest showcases multiple ways representative MAS methods can achieve a task. The framework filters 844 tasks requiring collaboration for instantiation. The paper also points out the shortcomings of current methods: outcome-only benchmarks overlook collaboration specifics, and LLM-as-Judge evaluations depend on model-specific inferences, which can vary. The proposed framework aspires to establish a standard for method evaluation. Authored by researchers, the paper is accessible on arXiv.