MP-ISMoE: Mixed-Precision Framework for Efficient Transfer Learning
A novel machine learning framework named MP-ISMoE (Mixed-Precision Interactive Side Mixture-of-Experts) has been introduced to tackle the challenges of memory and performance in parameter-efficient transfer learning. This method employs a Gaussian Noise Perturbed Iterative Quantization (GNP-IQ) technique to convert weights into lower bit representations, effectively minimizing quantization errors. The memory conserved through this process is redirected to bolster the learning capabilities of side networks. This strategy seeks to enhance efficiency while maintaining performance, specifically for downstream tasks in pre-trained foundation models. The research can be accessed on arXiv with the ID 2605.04058.
Key facts
- MP-ISMoE stands for Mixed-Precision Interactive Side Mixture-of-Experts.
- The framework includes a Gaussian Noise Perturbed Iterative Quantization (GNP-IQ) scheme.
- GNP-IQ quantizes weights into lower bits to reduce memory usage.
- Memory conserved from GNP-IQ is used to improve side network learning capacity.
- The method targets parameter-efficient transfer learning (PETL) and memory-efficient transfer learning (METL).
- The paper is available on arXiv with ID 2605.04058.
- The approach addresses memory overhead from gradient backpropagation.
- The framework aims to balance memory efficiency and performance.
Entities
Institutions
- arXiv