IMAX Framework Enhances Exploration in RLVR for LLM Reasoning
A novel approach known as Information-Maximizing Augmented eXploration (IMAX) tackles the issue of entropy collapse in reinforcement learning with verifiable rewards (RLVR) for large language models (LLMs). While RLVR enhances accuracy in single rollouts, it struggles to broaden coverage on effective reasoning paths due to sparse rewards and extended reasoning timelines. IMAX develops a set of soft prefixes that modify the base model's prior over reasoning paths, serving as adjustable control mechanisms to generate varied rollout distributions from the same foundational model. This method eliminates the need for reinforcement learning to promote exploration beyond the base model. The research can be found on arXiv under ID 2605.08817.
Key facts
- IMAX framework proposed for RLVR in LLM reasoning tasks
- Addresses entropy collapse phenomenon
- Uses pool of soft prefixes as trainable control knobs
- Induces distinct rollout distributions from same backbone model
- Avoids reliance on RL for exploration
- arXiv paper ID: 2605.08817
- Published on arXiv
Entities
Institutions
- arXiv