SORT: Selective Off-Policy Reference Tuning Improves Reasoning in LLMs

Researchers have unveiled Selective Off-Policy Reference Tuning (SORT), a technique that improves reinforcement learning by utilizing verifiable rewards for large language models. SORT tackles a significant drawback of GRPO-style approaches, which struggle with challenging prompts where every sampled rollout fails. Instead of altering the generation of rollouts, SORT implements a repair update: it formulates a plan based on the reference solution, evaluates token probabilities both with and without that plan, and prioritizes tokens that become more predictable when conditioned on the plan. This approach converts completely incorrect prompts into selective, structure-aware learning signals rather than simple imitation. SORT demonstrates enhancements over GRPO and guidance baselines across three backbones and eight reasoning benchmarks, particularly benefiting weaker models.