Multiverse Computing Cuts Knowledge Distillation Cost with Fused Chunked KL Loss
Multiverse Computing has published a new paper detailing an innovative technique aimed at improving memory efficiency in the knowledge distillation process for large language models (LLMs). This technique employs cached top-100 logits for offline distillation and utilizes a combined chunked KL divergence loss. As a result, maximum memory requirements have plummeted from 85.2 GiB to 5.45 GiB for processing 32K tokens. Furthermore, the distillation of the GPT-OSS 20B model can now be performed on a single GPU node, significantly speeding up the process and enhancing throughput. The model also upgrades from 8B to 3.2B parameters without losing benchmark accuracy.
Key facts
- Multiverse Computing released a paper on efficient knowledge distillation for LLMs.
- The paper introduces offline distillation using cached top-K logits.
- A fused chunked KL loss avoids materializing the full vocabulary × sequence matrix.
- Peak memory reduced from 85.2 GiB to 5.45 GiB at 32K tokens (15.6× reduction).
- Distillation of GPT-OSS 20B at 32K context shrank from four GPU nodes to one.
- Step time fell from 57.0 to 12.23 seconds (5× faster).
- Student model (3.2B params) retains most teacher accuracy on BoolQ and HellaSwag.
- Implementation open-sourced on GitHub.
Entities
Institutions
- Multiverse Computing
- Nvidia
- Hugging Face