Curriculum Learning-Guided Progressive Distillation for LLMs
A novel framework named Curriculum Learning-Guided Progressive Distillation (CLPD) tackles two often-neglected aspects of knowledge distillation in large language models: the sequence of training data and the disparity in capacity between teacher and student models. By creating a structured curriculum that arranges training examples from simpler to more complex, CLPD aligns the difficulty of data with the teacher's capabilities. Additionally, it implements an implicit curriculum that progressively introduces teachers with greater capacity. This method seeks to address the puzzling issue where more competent teachers do not yield superior students. CLPD is modular, allowing integration into current distillation processes. The findings are presented in arXiv paper 2605.11260.
Key facts
- CLPD stands for Curriculum Learning-Guided Progressive Distillation
- Addresses learning order of training data and capacity mismatch between teacher and student models
- Aligns data difficulty with teacher strength
- Constructs explicit curriculum from easy to hard examples
- Applies implicit curriculum by progressively scheduling teachers of increasing capacity
- Aims to resolve counter-intuitive phenomenon where stronger teachers fail to produce better students
- Framework is modular and integrable
- Paper available on arXiv with ID 2605.11260
Entities
Institutions
- arXiv