PODS: Down-Sampling Rollouts for Efficient LLM RL Training

PODS (Policy Optimization with Down-Sampling) has been developed by researchers to tackle the challenges of compute and memory imbalance in reinforcement learning with verifiable rewards (RLVR) for large language models. RLVR encounters a key issue: while rollout generation is highly parallel and requires minimal memory, policy updates demand significant communication and memory resources. By training exclusively on a carefully chosen subset of rollouts, PODS separates rollout generation from policy updates, preserving learning effectiveness and significantly lowering update expenses. The approach employs max-variance down-sampling, a methodical selection criterion that enhances reward diversity, and features an efficient O(n log n) implementation. In practice, Group Relative Policy Optimization (GRPO) utilizing PODS achieves the maximum test accuracy of standard GRPO at least 1.7 times faster across various tasks.