Covariance-Aware GRPO Improves LLM Reasoning by Taming Extreme Tokens

A new hyperparameter-free method, Covariance-Aware GRPO with Gaussian-Kernel Advantage Reweighting, addresses the exploration-exploitation tradeoff in Group Relative Policy Optimization (GRPO) for large language models. The approach dynamically down-weights extreme token-level updates using a Gaussian kernel, motivated by the theoretical insight that entropy changes are governed by covariance between token probabilities and advantages. Empirical evaluations show improved downstream performance across reasoning benchmarks compared to standard GRPO, while stabilizing entropy during training. The method is presented in a paper on arXiv (2605.11538) under Computer Science > Computation and Language.