Resets Improve Credit Assignment in Language Model Reasoning
A new arXiv preprint (2605.25507) proposes two methods—Random-Reset Policy Optimization (RRPO) and Self-Reset Policy Optimization (SRPO)—to improve credit assignment in reinforcement learning for language model reasoning. Current methods assign a single outcome reward uniformly across all tokens, ignoring which steps contributed to success or failure. RRPO draws reset states randomly from reasoning steps, while SRPO lets the model self-localize erroneous steps and reset there. Both enable more precise credit assignment by returning to intermediate states and resampling counterfactual continuations, allowing targeted refinement of faulty reasoning steps rather than updating entire trajectories uniformly.
Key facts
- arXiv:2605.25507
- RRPO draws reset states randomly from reasoning steps
- SRPO self-localizes erroneous steps and resets there
- Resets enable returning to intermediate states and resampling counterfactual continuations
- Uniform assignment ignores which steps contributed to success or failure
- Targeted refinement of faulty reasoning steps is enabled
- Contemporary reinforcement learning with verifiable reward methods post-trains language models on multi-step reasoning
- Outcome differences can be attributed to decisions made at the reset point
Entities
Institutions
- arXiv