Distribution Guided Policy Optimization for LLM Reasoning

A novel framework called Distribution Guided Policy Optimization (DGPO) has been introduced by researchers to enhance fine-grained credit assignment in reasoning with large language models, eliminating the need for a critic in reinforcement learning. DGPO overcomes the challenges faced by Group Relative Policy Optimization (GRPO), particularly in managing sequence-level credit assignment during extended Chain-of-Thought generations. The conventional unbounded Kullback-Leibler divergence penalty leads to gradient instability and a conservative approach that limits innovative reasoning paths. In contrast, DGPO views distribution deviation as a guiding signal instead of a strict penalty. This research has been submitted to arXiv (cs.LG) and can be accessed at https://arxiv.org/abs/2605.03327.