GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs

The newly introduced framework, GUI-Lens, seeks to enhance GUI grounding, which involves aligning natural-language commands with specific click points on a screen—crucial for effective GUI agents. This issue is particularly significant in high-resolution, cluttered interfaces where a vision-language model (VLM) might identify a desired control but struggle with accurate localization for interaction. Current techniques offer various localization aids but typically depend on direct click predictions, risking the propagation of visual ambiguities or imprecise initial estimates. GUI-Lens employs a coarse-to-fine grounding method, allowing a general-purpose VLM to identify targets through active visual observations. It extracts OCR text and UI component positions from screenshots as coordinate references. By utilizing the instruction, current view, and these references, the VLM iteratively refines its understanding to accurately identify the click location. This innovative approach, outlined in a paper on arXiv (arXiv:2608.03270), promises to enhance the accuracy of GUI agents, with implications for automated software testing, accessibility, and digital assistants.