Visual Tool-Use in Multimodal LLMs: A Causal Audit

A recent study published on arXiv (2608.06270) examines how effective visual tool-use is within multimodal large language models (LLMs), focusing on the 'thinking-with-images' approach that allows for actions such as crop-and-zoom. The research indicates that models using these techniques often show only slight or even negative improvements compared to direct inference, while also incurring significantly higher token expenses. Additionally, they tend to crop out irrelevant areas and struggle with questions that direct inference can answer accurately. To assess the causal impact of visual evidence on responses, the authors develop a causal graph distinguishing between observation-mediated paths and action-induced shortcuts. They evaluate this through interventions at three levels: policy, trajectory, and step. The step-level estimand, Visual Evidence Gain, helps to pinpoint the impact of each visual observation. This paper, authored by researchers, highlights potential limitations in current visual tool-use methods for multimodal AI systems.