VIGOR: Efficient Rollout Allocation for Reinforcement Learning
The introduction of a novel technique known as Variance Guided Online Rollout Allocation (VIGOR) enhances the effectiveness of reinforcement learning with verifiable rewards (RLVR) specifically for large language models. VIGOR tackles the high computational costs and inconsistencies associated with GRPO, which often produces numerous non-informative chain-of-thought rollouts. Rather than assigning a predetermined rollout budget for each example, VIGOR begins with a limited number of rollouts across all examples in a batch and progressively distributes extra rollouts based on variance assessments. This method not only shortens training duration but also boosts stability, all without the need for an expanded rollout pool or historical filtering.
Key facts
- VIGOR is a method for efficient rollout allocation in RLVR.
- It addresses computational expense and instability of GRPO.
- VIGOR starts with a small number of rollouts per example.
- It iteratively allocates additional rollouts based on variance.
- The method reduces training time and improves stability.
- VIGOR does not require a larger pool of rollouts or historical filtering.
- The research is published on arXiv with ID 2607.22002.
- GRPO relies on repeated generation of long chain-of-thought rollouts.
Entities
Institutions
- arXiv