SA-OPD: Filtering Spurious Signals in On-Policy Distillation
A recent study presents SA-OPD, a framework aimed at enhancing On-Policy Distillation (OPD) for language models. OPD facilitates the transfer of skills from a teacher model to a student by guiding student-generated trajectories with detailed token-level feedback. While recent selective OPD approaches focus on prioritizing signals according to confidence, informativeness, or learnability, they often neglect a critical issue: token-level evaluations may be influenced by general language biases, formatting norms, or stereotypical reasoning patterns instead of being based on task-specific data. The authors term this misleading yet optimization-relevant supervision 'spurious signals,' which can create significant gradients with minimal effect on task enhancement. To counter this, SA-OPD filters out misleading token-level guidance by assessing input-groundedness and its optimization effects. The paper can be found on arXiv with the identifier 2608.03632.
Key facts
- Paper introduces SA-OPD, a Spurious-Signal-Aware On-Policy Distillation framework.
- SA-OPD addresses a failure mode in language models where token-level judgments rely on input-agnostic priors.
- Spurious signals are defined as optimization-relevant but weakly input-grounded supervision.
- SA-OPD filters misleading supervision based on input-groundedness and optimization impact.
- The paper is published on arXiv with ID 2608.03632.
- The research focuses on improving On-Policy Distillation (OPD) for language models.
Entities
Institutions
- arXiv