Prune-OPD: Dynamic Rollout Truncation for Efficient Long-Horizon Reasoning Distillation
Prune-OPD is a framework introduced to address the inefficiency and reliability issues in on-policy distillation (OPD) for long-horizon reasoning tasks. OPD uses dense teacher rewards to improve student models, but as the student's generated prefix diverges from the teacher's thought process, the teacher's reward loses local exploitability, leading to degraded reward quality and computational waste. Prune-OPD dynamically aligns training budgets with supervision quality by continuously monitoring local compatibility between student and teacher predictions via top-k overlap. Upon detecting severe prefix-drift, it down-weights subsequent unreliable rewards and triggers dynamic rollout truncation, halting futile generation. The paper is available on arXiv under ID 2605.07804.
Key facts
- Prune-OPD is a framework for on-policy distillation in long-horizon reasoning.
- It addresses the problem of prefix-drift where student diverges from teacher.
- It uses top-k overlap to monitor local compatibility.
- Upon severe drift, it down-weights unreliable rewards and truncates rollout.
- The approach reduces computational waste.
- The paper is on arXiv: 2605.07804.
- It is a cross-type announcement.
- The method dynamically aligns training budgets with supervision quality.
Entities
Institutions
- arXiv