Token Credit in Self-Distillation: New Research from arXiv

A recent study published on arXiv (2608.09263) investigates the effectiveness of token credit assignment in on-policy self-distillation for large language models. The researchers contend that a change in token likelihood does not necessarily equate to credit for outcomes. They identify three critical inquiries: whether the score accurately reflects superior actions, how feedback construction alters comparisons, and what behavior the training loss promotes. The paper formally delineates these aspects. When scoring a rollout with hindsight feedback pertaining to the same rollout, the content influences both tokens and the scoring context, resulting in direct self-dependence. Utilizing feedback from a different rollout of the same issue eliminates this dependence, yet does not ensure a meaningful score. In experiments with a 20B model on AIME 2025, the additive score achieved is close to chance (AUC=0.505) and slightly favors incorrect tokens. This study underscores the necessity for thorough validation of privileged self-distillation techniques.