VAKRA: New Benchmark Evaluates Multi-Hop Reasoning Across APIs and Retrieval

A new benchmark called VAKRA (Valuating API and Knowledge Retrieval Agents) has been developed to assess the abilities of AI agents in business environments where reasoning across structured APIs and document collections is essential. Unlike existing benchmarks that evaluate these skills separately, VAKRA integrates them, offering over 8,000 executable APIs from 62 different domains. It presents tasks in three progressively challenging scenarios: varied API interaction methods, multi-hop reasoning with structured APIs, and reasoning from multiple sources under natural-language tool-use policy constraints. The accuracy of predictions is confirmed by re-executing tool calls on live APIs, allowing for various valid outcomes. Evaluations of frontier and open-weight models using a fixed ReAct harness show that the top model only reaches 70.4% accuracy on single-hop endpoint tasks and falls to 50–51% for compositional APIs, highlighting the benchmark's goal of realistically measuring agent performance in complex, real-world situations.