AntiSD: New RL Method Reverses Self-Distillation for Math Reasoning

A new reinforcement learning method called Anti-Self-Distillation (AntiSD) is proposed to improve reasoning in large language models, specifically for math tasks. The approach addresses failures in on-policy self-distillation, where a student model learns from a copy of itself conditioned on privileged context like verified solutions. Using pointwise mutual information analysis, researchers found that privileged context inflates teacher confidence on structural tokens (e.g., connectives, verifiable claims) and deflates it on deliberation tokens (e.g., 'Wait', 'Let', 'Maybe') crucial for multi-step search. AntiSD reverses the divergence direction, ascending rather than descending between student and teacher, yielding a naturally bounded advantage per token. An entropy-triggered gate disables the term when teacher entropy collapses. The method is detailed in arXiv paper 2605.11609, authored by researchers from an unspecified institution.