PluginEval: New Benchmark for Fine-Grained Error Attribution in Tool Routing

A new diagnostic benchmark called PluginEval has been developed by researchers to assess fine-grained error attribution in function calling by large language models (LLMs). This benchmark tackles three key structural issues in existing evaluation techniques: the prevalence of power-law data distributions that fail to adequately represent rare cases, the lack of adversarial hard negatives that mask performance disparities among models, and the dependence on unvalidated judgments from LLMs in annotation processes. PluginEval utilizes a two-stage approach: initially, it treats tool routing as a sequence of three decisions, distinguishing generation from verification, where LLMs suggest candidate calls and deterministic validation along with actual API execution offers dependable quality indicators. Subsequently, it breaks down each plugin by capability, intent, and boundary to pinpoint trigger and exclusion scenarios, generating queries at a detailed level. The benchmark seeks to enhance the reliability of tool routing as LLMs take on more autonomous roles. The related paper can be found on arXiv with the identifier 2608.08700.