AppDeltaWorld: A New AI Model for Mobile GUI Agents
A new AI model named AppDeltaWorld has been developed by researchers to enhance mobile GUI agents by predicting transitions in GUI as code updates instead of relying on images or text. This model tackles the difficulties associated with gathering real-world interaction data for sensitive applications and privacy-sensitive tasks, as well as the expensive nature of simulated environments. AppDeltaWorld extracts app-specific Level-1 HTML references based on action-transition constraints, creates Level-2 executable HTML based on the current screen, action, anticipated next-screen text, and retrieved structure, and integrates the generated code to form the next GUI state. This methodology seeks to improve stability, coverage of modalities, and consistency in action-transition within GUI world models. The research paper can be found on arXiv with the identifier 2608.05891.
Key facts
- AppDeltaWorld is a transition-grounded delta code world model for mobile GUI agents.
- It predicts the next GUI as a code update instead of an image or text.
- It retrieves app-specific Level-1 HTML references under an action-transition constraint.
- It generates Level-2 executable HTML conditioned on current screen, action, predicted next-screen text, and retrieved structure.
- The model addresses limitations in existing GUI world models: unstable generation, limited modality coverage, and inconsistent action-transition logic.
- Real trajectories for sensitive apps are difficult to obtain.
- Simulated environments are costly to scale up.
- The paper is available on arXiv with ID 2608.05891.
Entities
Institutions
- arXiv