ReCo Method Mitigates GRPO Distributional Concentration in LLM Training

A recent study published on arXiv (2607.26862) highlights a weakness in Group Relative Policy Optimization (GRPO), a common reinforcement learning technique used for post-training language models. The researchers discovered that GRPO diminishes a base model’s reasoning abilities and performs poorly in Pass@k when k is large, suggesting a decrease in reasoning path coverage. This issue arises because GRPO focuses on responses that the base model generates with high likelihood. Two factors contribute to this: at the response level, frequently occurring high-probability responses skew the group gradient; at the token level, GRPO's importance ratio amplifies gradients, favoring tokens that are more probable under the current policy. To counter this, the authors introduce ReCo, a reweighting strategy that normalizes response contributions based on their expected frequency within the rollout group, addressing the token-level importance ratio. The paper is a cross submission aimed at enhancing the diversity and coverage of reasoning paths in LLM post-training.