DVPO: Distributional Value Modeling for Robust LLM Post-Training

A new reinforcement learning framework, DVPO (Distributional Value Modeling with Risk-aware Policy Optimization), has been introduced to improve LLM post-training under noisy or incomplete supervision. The approach combines conditional risk theory with distributional value modeling to balance robustness and generalization. DVPO learns token-level value distributions for fine-grained supervision and applies asymmetric risk regularization to shape the distribution. It addresses limitations of existing methods like worst-case optimization (RFQI, CQL) and mean-based approaches (PPO, GRPO), which can be overly conservative or uneven across scenarios. The paper is available on arXiv under ID 2512.03847.