Critic-Free Pretraining Boosts Online RL Fine-Tuning Efficiency
A novel machine learning technique known as Critic-Free Pretraining (CFP) has been unveiled to enhance the effectiveness of offline-to-online (O2O) reinforcement learning. This method, outlined in a paper submitted to arXiv on August 26, 2025 (arXiv ID 2608.10473), tackles a significant issue in O2O RL: the direct reuse of an offline-trained critic can impede online fine-tuning. As the policy and data distribution change swiftly during online interactions, offline value estimates may misalign with the online context, resulting in ineffective policy enhancement and exploration. CFP eliminates the need for offline critic training, enabling a newly initialized critic to adjust without biased estimates. It is compatible with various mainstream O2O algorithms and often surpasses traditional O2O methods across a range of tasks, especially on more challenging ones. The paper includes tools for citation, code, and data and is part of the arXivLabs initiative, promoting collaborative experimental efforts.
Key facts
- Critic-Free Pretraining (CFP) is a new method for offline-to-online reinforcement learning.
- CFP abandons offline critic training to avoid biased value estimates.
- It allows a freshly initialized critic to adapt during online fine-tuning.
- CFP is compatible with various mainstream O2O algorithms.
- It consistently matches or improves upon conventional O2O algorithms.
- Particularly pronounced gains are observed on several challenging tasks.
- The paper is available on arXiv with ID 2608.10473.
- Submitted to the Computer Science > Machine Learning category.
Entities
Institutions
- arXiv