MissClick: New Attack Exploits Digit-Serialized Coordinates in GUI Grounding Models

A recent paper titled 'MissClick: Exploiting Digit-Serialized Coordinates to Attack GUI Grounding Models' has been published on arXiv (ID: 2608.03740). This study uncovers a security flaw in GUI visual grounding models, which translate screen coordinates into actionable clicks. These models output coordinates as sequences of digit tokens that are converted into numerical values. The researchers note that each digit is predicted as a categorical token, and altering the hundreds-place digit by one results in a 100-unit shift in the coordinate, potentially leading to significant click displacement. The paper outlines attack strategies that consider the numerical and place-value characteristics of coordinate outputs. It differentiates between untargeted and targeted attacks, which aim to displace clicks outside the correct area or into a designated region, respectively. The MissClick attack is characterized as straightforward and effective, with relevance to the security of AI-based interfaces.