Entropy-Scaled Trust Regions for Asynchronous RL
A new arXiv paper (2607.22186) reveals that importance ratios in asynchronous reinforcement learning scale systematically with token entropy, causing distinct phenomena at low and high entropy. At low entropy, train-inference discrepancy amplifies into sampling noise; at high entropy, weight updates create legitimate exploratory deviations. Existing methods that apply uniform thresholds based solely on ratio magnitude inadvertently admit amplified noise while strictly limiting exploration. The paper proposes entropy-scaled trust regions to address this, improving stability in LLM post-training.
Key facts
- arXiv paper 2607.22186
- Asynchronous RL used for LLM post-training
- Stale off-policy data can cause policy collapse
- Importance ratio scale varies with token entropy
- Low entropy amplifies train-inference discrepancy into noise
- High entropy induces legitimate exploratory deviations
- Magnitude-only correction admits noise and limits exploration
- Entropy-scaled trust regions proposed as solution
Entities
Institutions
- arXiv