Tail-Aware Top-k On-Policy Distillation Improves Language Model Training

A novel approach named Tail-Aware Top-k On-Policy Distillation (TA-OPD) has been developed by researchers to enhance the training of language models, specifically targeting shortcomings in current on-policy distillation methods. Traditional on-policy distillation (OPD) enables a student model to replicate the next-token distribution of a teacher model. However, many existing techniques focus solely on minimizing the reverse Kullback-Leibler (KL) divergence concerning the teacher's top-k tokens, neglecting tail probability information. This neglect leads to increased tail probability and entropy in the student model, resulting in reduced accuracy in downstream tasks. TA-OPD rectifies this by minimizing reverse KL divergence over the top-k tokens while incorporating tail information. Details of this method can be found in a paper on arXiv (arXiv:2608.14728), aiming to enhance knowledge distillation efficiency in large language models.