RobustBench-TC: Benchmarking Sim-to-Real Gap in Tool-Use Language Agents

A new benchmark, RobustBench-TC, exposes the sim-to-real gap in tool-use language agents by introducing 22 perturbation types grounded in real-world failures from GitHub issues. The perturbations are organized by four components of the tool-use POMDP: observation, action space, reward-relevant metadata, and transition dynamics. Testing across 21 models from 1.5B to 32B parameters, including the closed-source o4-mini, reveals sharply uneven robustness: observation perturbations reduce accuracy by less than other types. The work highlights that current benchmarks assume clean inputs, unambiguous tool registries, and reliable APIs, while real deployments suffer from user typos, misconfigured timeouts, and duplicate tool names.