Dyna-Style RL Improves Quadrupedal Locomotion Sample Efficiency

A novel reinforcement learning strategy for quadrupedal movement, outlined in a paper on arXiv (2509.06296), employs model-based methods to enhance sample efficiency. This Dyna-style approach supplements PPO rollouts with synthetic data produced by a learned transition model, utilizing short-horizon synthetic tails anchored by physics-based simulations for stability. A scheduled strategy gradually incorporates synthetic transitions, minimizing model reliance during the early training phase when prediction accuracy is low. Comprehensive ablation studies investigate the impact of various data parameters on PPO's learning dynamics. Tested on a Unitree Go1 robot, the method achieved convergence with significantly fewer simulation steps (19.64M). This work addresses the inefficiencies of traditional on-policy RL controllers, which often demand millions of interactions. The research team announced this replace-cross type on arXiv, potentially leading to more efficient training for legged robots, thus lowering computational costs and time.