Hybrid Policy Distillation Optimizes LLM Compression

A recent study published on arXiv introduces Hybrid Policy Distillation (HPD) aimed at compressing large language models (LLMs). This innovative approach merges forward and reverse KL divergence to effectively balance mode coverage with mode-seeking, utilizing off-policy data alongside efficient on-policy sampling. HPD has been tested on various tasks, including long-generation math reasoning, short-generation dialogue, and coding challenges, demonstrating enhanced optimization stability, computational efficiency, and overall performance across different model families and sizes. The associated code can be found at the specified URL. Additionally, the paper offers a cohesive perspective on knowledge distillation, framing it as a reweighted log-likelihood objective at the token level.