State Transition Pretraining Boosts GUI Agent Performance

A novel approach called State Transition Pretraining (STP) has been introduced by researchers, serving as a new scaling axis for GUI agents through the ongoing pretraining of a unified multimodal model focused on visual state transitions. This technique simultaneously enhances inverse dynamics (which involves predicting actions based on state changes) and forward dynamics (which entails forecasting subsequent states from current states and actions). As a result, the model gains action-grounded visual representations and a comprehensive internal world model of GUI dynamics. When fine-tuned with task trajectories, models trained with STP surpass baseline performance on desktop and mobile benchmarks, such as AgentNetBench, AndroidControl, and GUIOdyssey. Studies indicate that optimizing joint dynamics leads to consistent gains compared to single-objective training, with performance improving as pretraining data volume increases. The findings are published in arXiv preprint 2607.24112.