SimpleOPD: Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning

Researchers have introduced SimpleOPD, a technique for on-policy distillation (OPD) aimed at transferring reasoning abilities from long-context teacher models to short-context student models. This addresses issues such as tokenizer mismatch, distribution discrepancies, response length inflation, and training instability. The method showcases the transfer of proof-reasoning skills from the long-context model SU-01 to short-context students. To resolve tokenizer differences, OPD is executed within a shared text space, aligning tokens that occupy the same text spans across both tokenizers. Additionally, a student reference KL loss is implemented to manage excessive generation length and frequent truncation, while masking the benefits of special termination tokens (e.g., <|im_end|>) to prevent the student from deviating too far from its original distribution. The paper can be found on arXiv under identifier 2608.14277.