SMOPD: Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation
A novel technique known as SMOPD (Specialize-and-Merge Online Policy Distillation) has been introduced to enhance multi-reward reinforcement learning. This method seeks to overcome the shortcomings of the existing Group reward-Decoupled Normalization Policy Optimization (GDPO), which separately normalizes each reward dimension prior to aggregation to reduce reward masking. Nevertheless, experiments indicate that GDPO continues to face challenges in balancing reward signals with varying granularities, particularly when integrating dense rewards (fine-grained scores ranging from 0.1 to 1.0) with sparse rewards (binary 0 or 1). In these scenarios, the sparse reward may lack sufficient optimization signals, hindering effective reinforcement. SMOPD is designed to bolster the optimization signal from sparse rewards while maintaining the effectiveness of dense rewards. This method is elaborated in a paper available on arXiv (arXiv:2608.03092v1), authored by researchers proposing this innovative approach to improve multi-reward training.
Key facts
- SMOPD is a new method for multi-reward reinforcement learning.
- It addresses limitations of GDPO (Group reward-Decoupled Normalization Policy Optimization).
- GDPO normalizes each reward dimension separately before aggregation.
- Experiments show GDPO struggles with reward signals of different granularities.
- Dense rewards assign fine-grained scores (0.1 to 1.0), while sparse rewards provide binary feedback (0 or 1).
- Sparse rewards may provide insufficient optimization signal.
- SMOPD aims to strengthen sparse reward signals without sacrificing dense reward capabilities.
- The paper is available on arXiv with ID 2608.03092v1.
Entities
—