QLPO: Length-Aware Policy Optimization via Resampling

A novel technique known as Quadrant-weighted Sampling for Length-aware Policy Optimization (QLPO) tackles the issue of lengthy chain-of-thought outputs in substantial reasoning models during reinforcement learning. QLPO, a resampling-based adaptation of GRPO, introduces implicit control over response length without altering the reward function. Initially, it generates an excess of candidate responses and subsequently resamples the training set, maintaining the empirical ratio of correct to incorrect answers while promoting shorter correct responses and longer incorrect ones. This approach modifies the training distribution and subtly encourages the generation of shorter outputs from the model. The effectiveness of this method was evaluated on models with parameter counts ranging from 1.5B to 32B, encompassing both base and advanced reasoning models.