UI-Mate: Open-Weight GUI Agent with In-Context Demonstrations
A new foundation GUI agent named UI-Mate has been developed by researchers to streamline complex digital tasks through an integrated training stack that combines environment-grounded techniques with in-context demonstration learning. This innovative system tackles issues such as limited and biased training data, unclear prompts, and inconsistent execution. UI-Mate's training stack features a closed-loop data engine that automates various processes, including task generation, environment setup, rollout, filtering, capability balancing, supervised fine-tuning (SFT), and online reinforcement learning within massively parallel environments using unified task-verifier bundles. Its in-context demonstration learning converts multimodal demonstrations into adaptable subtask workflows, allowing the agent to adhere to relevant steps and re-plan from the live interface. Detailed in a paper on arXiv (2608.15930), the agent is referred to as OSWorker, with the submission emphasizing its potential for enhanced GUI automation reliability and adaptability.
Key facts
- UI-Mate is a foundation GUI agent for automating complex digital tasks.
- It integrates environment-grounded training with in-context demonstration learning.
- The training stack includes a closed-loop data engine for task generation and environment construction.
- It uses unified task-verifier bundles for rollout, filtering, and capability balancing.
- In-context demonstration learning transforms multimodal demos into subtask-level workflows.
- The agent can re-plan from the live interface.
- The paper is available on arXiv with ID 2608.15930.
- The agent is named OSWorker.
Entities
Institutions
- arXiv