TASTE: Automated Method Generates Challenging AI Agent Benchmarks

A new paper on arXiv introduces TASTE (Task Synthesis from Tool Sequence Evolution), an automatic method for generating challenging benchmarks for AI agents. As existing benchmarks like τ²-Bench become saturated, TASTE reverses the traditional task construction process by first evolving tool sequences and then instantiating them into tasks. It uses an Adaptive Contrastive n-gram model trained on LLM-judged validity signals to sample valid tool sequences with broad coverage. Representative sequences are selected via clustering, then refined into complete benchmark tasks. The method addresses the high cost and complexity of manual benchmark creation and expands the range of tool-use patterns tested.