CVPO: New RL Method Enhances LLM Reasoning with Value-Variance Adaptation

A novel reinforcement learning technique called Curriculum-guided Value-Variance Policy Optimization (CVPO) has been introduced to enhance the reasoning skills of large language models (LLMs). This method tackles two major challenges in current RL strategies: the lack of precise feedback on answer trajectories and the issue of varying problem difficulty. CVPO functions on two levels: it employs token-level value-variance at the response trajectory level to measure intrinsic randomness, creating a variance-aware advantage adjustment for various reward types. At the question level, it features a dynamic curriculum weighting system that adjusts based on question difficulty, aligning tasks with the model's abilities. The paper, found on arXiv (2608.03068), includes a theoretical analysis indicating that value-variance constrains policy update magnitude and suggests a link between variance and exploration intensity, potentially improving RL training efficiency for LLMs.