Graph-Based Difficulty Estimator Improves RLVR Rollout Scheduling

Reinforcement learning with verifiable rewards (RLVR) enhances the reasoning abilities of large language models but entails expensive rollout exploration. Present adaptive schedulers rely on curriculum-based selection or uneven rollout distribution, yet they often fail to provide accurate online difficulty assessments. Probing techniques introduce additional overhead, while estimators based on historical data suffer from cold starts and outdated feedback. This study introduces a graph-based online difficulty estimator that disseminates rollout feedback across various samples, improving difficulty evaluation without incurring probing costs. It seamlessly integrates into current RLVR training frameworks, optimizing exploration toward samples that require it most. This approach tackles the shortcomings of uniform budget allocation and existing scheduling methods, potentially reducing computational costs while boosting reasoning skills. The paper can be found on arXiv under identifier 2608.17941.