Masked Distillation Internalizes Chain-of-Thought in Language Models

A novel framework for knowledge distillation, known as masked distillation, allows student language models to grasp the reasoning processes of larger models. Large Reasoning Models (LRMs) produce detailed sequences of intermediate tokens prior to arriving at final answers, which significantly impacts both latency and expenses; however, the correctness of these traces does not necessarily correlate with the accuracy of the answers. In masked distillation, a student LLM learns to anticipate only the solution tokens based on the question, while a reasoning teacher offers guidance through its own chain-of-thought. This method seeks to minimize the length of intermediate traces and reduce operational costs while maintaining answer precision. The research is available on arXiv with ID 2607.22629.