Activation-Guided Pruning Enables Cross-Scale Model Fusion
A new study on arXiv (2608.13596) introduces a method called Activation-Prune-Merge (APM) that allows knowledge transfer from a large language model to a smaller one without any training. This approach creates specific activation maps from the larger model, pinpoints crucial layers, hidden dimensions, attention heads, and MLP neurons, and then prunes them to match the smaller model’s structure. It also uses a micro interpolation coefficient in the final output. This technique boosts the smaller model's performance without needing direct neuron-wise alignment. The authors emphasize the importance of addressing cross-scale scenarios in model fusion, where models differ in various aspects. They also mention that simply reducing a large model and incorporating a small mixing weight can benefit the smaller model, which APM utilizes.
Key facts
- Paper arXiv:2608.13596 introduces Activation-Prune-Merge (APM).
- APM is a training-free framework for cross-scale model fusion.
- It transfers knowledge from a large donor model to a smaller recipient.
- Method uses activation maps to prune donor to recipient architecture.
- Injects donor slice with micro interpolation coefficient.
- No explicit neuron-wise semantic alignment is required.
- Addresses heterogeneous model fusion with architectural mismatch.
- Paper is available on arXiv.
Entities
Institutions
- arXiv