ScreenSearch: AI System for Desktop GUI Exploration
Researchers have created a new tool called ScreenSearch aimed at improving how we explore operating system interfaces with uncertainty in mind. Desktop GUI agents operate under partial visibility, meaning that similar-looking screens might actually indicate different workflow states, leading to different results. ScreenSearch combines a method for retrieving and deduplicating screens with a smart algorithm that helps in exploring desktops more effectively. It converts UIA trees into specific features, indexes screens using sparse token searches, and maintains a shared state graph among virtual machine workers. An uncertainty signal, based on the variety of outcomes, helps the system decide when to dive deeper or stick with what it knows. This study was shared on arXiv under the identifier 2605.16024.
Key facts
- ScreenSearch addresses partial observability in desktop GUI agents.
- Uses structural screen retrieval and deduplication.
- Employs ambiguity-aware PUCT graph-bandit algorithm.
- Converts UIA trees into location-aware structural features.
- Indexes related screens via sparse token search and metadata filters.
- Maintains shared deduplicated state graph across VM workers.
- Defines ambiguity signal based on matched-action outcome dispersion.
- Published on arXiv with ID 2605.16024.
Entities
Institutions
- arXiv