Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and Fused Chunked KL Loss
A recent study published on arXiv (2608.03796) explores how to enhance the efficiency of knowledge distillation (KD) training for smaller language models. The researchers make two key contributions. Firstly, they show that offline KD, which involves caching the teacher's top-K logits for training the student, achieves comparable training loss to online distillation while being quicker and more memory-efficient. It operates approximately 29% faster per iteration and can deliver up to 41% greater throughput on a single H200 GPU, eliminating the need for the teacher in memory. Secondly, they present a fused, chunked KL loss that circumvents the need to create a full vocabulary-sized logit tensor, resulting in peak memory usage that scales linearly with sequence length. This method tackles the significant costs associated with distillation, which often impacts the quality of compressed models. The paper is classified as a cross-type announcement and can be accessed at https://arxiv.org/abs/2608.03796.
Key facts
- Paper arXiv:2608.03796 presents efficient knowledge distillation for LLMs.
- Offline KD caches teacher's top-K logits once and trains student against cache.
- Offline KD matches online distillation in training loss.
- Offline KD runs about 29% faster per iteration.
- Offline KD achieves up to 41% higher throughput on a single H200 GPU.
- Offline KD removes teacher from memory.
- Fused, chunked KL loss avoids materializing full vocabulary-sized logit tensor.
- Peak memory becomes linear in sequence length.
Entities
Institutions
- arXiv