SP3O: New Preference-Based RL Algorithm Eliminates Reward Modeling
A new reinforcement learning algorithm, Segment Pairwise Proximal Policy Optimization (SP3O), has been introduced in a paper on arXiv (2608.02951). The method is reward-model-free, critic-free, and gradient-based, designed to work with segment preferences rather than trajectory-level feedback. This addresses limitations of existing preference-based RL methods, which either require reward models, are restricted to bandits or deterministic MDPs, or use slower zeroth-order optimization. SP3O leverages segment-level preference feedback to construct an accurate policy value difference, making it more efficient and easier for human evaluators to provide feedback on shorter segments. The paper is authored by researchers and was announced as a cross-type submission. The algorithm is significant for advancing preference-based RL in stochastic environments, potentially improving applications in AI training and robotics.
Key facts
- SP3O is a novel reinforcement learning algorithm.
- It is reward-model-free, critic-free, and gradient-based.
- It uses segment preferences instead of trajectory-level feedback.
- The method is compatible with general stochastic MDPs.
- Existing methods like DPO and P3O are limited to bandits or deterministic MDPs.
- Zeroth-order methods exhibit slower convergence rates.
- Segment preferences are easier for human evaluators to compare.
- The paper is available on arXiv with ID 2608.02951.
Entities
Institutions
- arXiv