CoCaRS: Correlation Calibration for Heterogeneous Knowledge Distillation
A novel approach known as CoCaRS (Correlation Calibration-Based Redundancy Suppression) has been introduced to enhance heterogeneous knowledge distillation. Knowledge distillation (KD) facilitates the training of a compact student model by leveraging a powerful teacher, which is essential for model compression. While KD has evolved from homogeneous to heterogeneous environments due to varying model architectures, discrepancies in architectural inductive biases hinder effective knowledge transfer. The concept of redundancy suppression aims to maintain cross-architecture invariance and minimize feature redundancy by decorrelating teacher-student feature correlations. Nonetheless, uniform decorrelation may diminish valuable structural information, and a static coefficient can make redundancy suppression's effectiveness dependent on specific teacher-student pairs and training phases. CoCaRS resolves these challenges by calibrating correlations to suppress redundancy while preserving structural integrity. Further details can be found in arXiv:2607.27054.
Key facts
- CoCaRS stands for Correlation Calibration-Based Redundancy Suppression.
- It addresses heterogeneous knowledge distillation.
- Knowledge distillation enables a compact student model to learn from a powerful teacher.
- Heterogeneous KD involves teacher and student models with different architectures.
- Architectural inductive biases cause representation discrepancies.
- Redundancy suppression preserves cross-architecture invariance.
- Uniform decorrelation may weaken useful structural information.
- A fixed coefficient makes redundancy suppression sensitive to teacher-student pairs and training stages.
Entities
Institutions
- arXiv