Hybrid GUI-MCP Agents: Tool Adoption Gap and RL Solutions

A recent paper on arXiv (2608.03327) explores hybrid agents capable of utilizing both screenshots and text tools. The research indicates that simply providing tools does not ensure their effective application. Utilizing the same GUI-MCP framework on the OSWorld-MCP benchmark (309 tasks), the MCP tools enhanced a reasoning model's accuracy by +4.0 percentage points, while a non-reasoning model's performance decreased by -5.9 points (5 runs each, both exceeding 2 standard errors). The distinction lies in tool decision-making: the non-reasoning policy misuses or overlooks tools, while the reasoning model, despite avoiding these errors, only engages a tool in 55 out of 309 tasks, representing a 23.9% adoption rate. This discrepancy is referred to as the 'adoption gap.' Both issues stem from the model's tendency to favor a less complex approach, as it is not trained to utilize tools. Multi-turn reinforcement learning (RL) strategies address this issue, with a dense tool bonus increasing spreadsheet usage from 0.03 to 0.33, although overall accuracy remains unchanged. The findings are summarized in the paper's abstract, marking its new submission on arXiv.