Knowledge Distillation Must Account for What It Loses
A new paper on arXiv critiques how we currently assess knowledge distillation. It argues that the focus on just task scores misses some critical losses in capabilities. The authors highlight a 'retention assumption' that exists in the metrics we use and suggest that distillation naturally leads to losing certain teacher behaviors. They categorize various off-metric losses, including aspects like uncertainty, boundary behavior, and diversity. To address these shortcomings, they propose specific targets for what should be preserved in different scenarios and introduce a concept called the Distillation Loss Statement. If you want to dive deeper, you can find the full paper at arXiv:2604.25110.
Key facts
- arXiv:2604.25110
- Position paper on knowledge distillation
- Argues student models should be judged by preservation of teacher capabilities
- Identifies retention assumption in current evaluation
- Reframes distillation as lossy projection
- Taxonomy of off-metric losses includes uncertainty, boundary behavior, process reliability, on-policy stability, grounding, privacy, safety, diversity
- Proposes scenario-specific preservation targets
- Proposes Distillation Loss Statement
Entities
Institutions
- arXiv