Canary Tools Diagnose Tool-Selection Reasoning in LLM Agents

So, there's this new study that introduces something called 'canary tools' to help identify where large language model (LLM) agents go wrong in choosing tools. You can find the study on arXiv with the code 2608.04719. They classify tool-selection mistakes into six categories, like semantic decoys and granularity traps, which helps to analyze issues more deeply than just saying a tool was wrong. They tested eight models—six hosted and two 8B open-weight—over 120 tasks, leading to a total of 8,640 runs. They also had a smaller test of 2,880 runs to look at subtlety. An independent evaluator reviewed the results, backed by another judge. The findings suggest that better models are less likely to fall for these traps, which is a big deal for improving LLM assessments in AI.