CUADebug: A Framework for Diagnosing Computer-Use Agent Failures

A new framework named CUADebug has been developed by researchers to identify and resolve issues in computer-use agents (CUAs), which interact with actual desktop and web interfaces using screenshots, keyboard and mouse inputs, and stateful UI feedback. Unlike agents that rely solely on text, CUAs encounter unique failures due to challenges in visual perception, spatial grounding, low-level interactions, task reasoning, and environmental changes, making debugging a complex multimodal causal localization issue. CUADebug features a specific error taxonomy for CUAs, a human-annotated benchmark called CUAErrorBench, and a tool-enhanced debugger known as CUADebugger. Instead of analyzing the entire process at once, CUADebugger scrutinizes questionable steps using paired before/after screenshots and action traces, ultimately providing a structured diagnosis that includes the root cause, error type, supporting evidence, and suggested corrective actions. This framework is elaborated in a paper on arXiv (arXiv:2608.02643) and is recognized as a cross-type submission, filling a significant gap in the debugging of multimodal agents and providing a methodical way to detect and rectify failures in real-world interface interactions.