SMOPD: Stabilizing Multi-Turn Self-Distillation via Token Entropy Masking

A novel technique named SMOPD (Selective Masking for On-Policy Distillation) has been developed to tackle the instability issues in multi-turn on-policy self-distillation (OPSD) for language models. This instability stems from 'dirty-history' rollouts, where incorrect intermediate responses influence later turns, and uniform distillation can misallocate loss to tokens that provide minimal corrective feedback. SMOPD functions as a loss-only stabilization method, prioritizing token positions based on student entropy for each middle-turn reply, discarding the lowest-entropy 20% from the clipped generalized Jensen-Shannon distillation loss, while maintaining the final-answer and FULL-preservation losses. This technique focuses on token-level uncertainty, does not introduce additional parameters, and incurs no inference-time overhead. It was evaluated against a correctness-scaling variant that enhances a common detached reliability proxy based on final-answer accuracy. On the LiC benchmark using Qwen3 models, SMOPD enhances SHA (the metric is truncated in the abstract). The research is accessible on arXiv with the identifier 2608.14647, categorized as 'cross'.