TMRL: Diffusion Timestep-Modulated Pretraining for Efficient Robot Policy Finetuning

The newly introduced Timestep-Modulated Reinforcement Learning (TMRL) framework seeks to enhance the fine-tuning efficiency of pre-trained robot policies through reinforcement learning (RL). A prevalent issue is that pre-training via behavioral cloning (BC) often leads to limited action distributions, hindering exploration during subsequent RL fine-tuning. To tackle this, the method known as Context-Smoothed Pre-training (CSP) adds forward-diffusion noise to policy inputs, facilitating a spectrum between accurate imitation and extensive action diversity. TMRL then fine-tunes these pre-trained policies by adjusting the diffusion timestep, allowing for better exploration control. This framework works well with various policy inputs, including states and 3D point clouds. The research, identified as arXiv:2605.12236 and categorized as 'replace-cross', is expected to attract interest from robotics and RL researchers.