REOPD: Adaptive Reward Extrapolation for On-Policy Distillation

A novel framework known as REOPD (Reliability-Adaptive Reward Extrapolation for On-Policy Distillation) has been introduced to tackle the issues of reward hacking and training instability in on-policy distillation (OPD). In OPD, a student model is trained using its own trajectories, benefiting from dense token-level supervision from a teacher. Current reward-extrapolation techniques, such as ExOPD, enhance the teacher-reference log-likelihood ratio to transcend mere imitation; however, they utilize a single global coefficient lambda for all tokens, potentially leading the student to conform to extreme peaks in implicit rewards, resulting in instability and reward hacking. Additionally, the optimal lambda varies by domain, necessitating expensive hyperparameter tuning. REOPD presents a token-level compatibility weight along with a batch-level adaptive budget, resulting in a token-wise coefficient lambda_{b,t} = 1 + gamma_b q_t. This method maintains alignment with the teacher while selectively extrapolating in reliable directions. Notably, REOPD does not require a verifier, reward model, value model, or additional rollout beyond standard OPD. The research paper can be found on arXiv with the identifier 2608.11698 and was released as a cross-type submission.