R²VPO: A New Reinforcement Learning Method Without Clipping

A new reinforcement learning method called Ratio-Variance Regularized Policy Optimization (R²VPO) eliminates the need for heuristic clipping in on-policy algorithms. Standard on-policy RL uses clipping to enforce trust regions, but this truncates high-return updates. R²VPO constrains policy ratio variance as a principled local approximation, acting as a distributional soft brake that preserves gradient signals from novel discoveries and enables reuse of stale off-policy data. The method is implemented via a primal-dual optimization framework. Evaluations across 7 LLM scales (fast and slow reasoning) and 10 robotic control tasks demonstrate generality.