AndroidReality: Benchmarking Mobile Agents' Robustness in Real-World Conditions
A new framework named AndroidReality has been developed by researchers to assess and enhance the resilience of mobile agents. This framework tackles the notable decline in performance that these agents face during real-world applications, in contrast to pristine online standards such as AndroidWorld. Utilizing a Markov Decision Process (MDP) approach, the researchers categorized real-world interface variability into a taxonomy of perturbations across three dimensions: state, transition, and action. This classification facilitated the creation of a perturbed mobile benchmark based on AndroidWorld, incorporating realistic and manageable perturbation injections for thorough robustness testing. The findings uncovered significant robustness deficiencies and highlighted four common error types, leading to the introduction of a straightforward, training-free Test-Time Introspective Recovery (TTIR) method to address these challenges. The comprehensive study is documented in a paper on arXiv (arXiv:2608.07775).
Key facts
- AndroidReality is a perturbation-based framework for evaluating mobile agent robustness.
- The framework uses a Markov Decision Process (MDP) perspective to categorize perturbations.
- Perturbations are organized along three axes: state, transition, and action.
- The benchmark is built on top of AndroidWorld with realistic perturbation injections.
- Evaluation showed substantial robustness gaps and four recurring error categories.
- A training-free Test-Time Introspective Recovery (TTIR) mechanism was proposed.
- The paper is available on arXiv with ID 2608.07775.
- The work aims to bridge the gap between clean benchmarks and real-world deployment.
Entities
Institutions
- arXiv