TriSP: Efficient Structured Pruning for Large Language Models
A novel importance metric for the structured pruning of large language models (LLMs), named TriSP (Tri-Signal Structured Pruning), has been introduced by researchers. This technique focuses on eliminating entire components, such as attention heads and MLP neurons, to develop more compact and efficient models. Current methods rely on either memory-heavy gradient-based estimations or activation-based proxies that fail to directly assess loss impact. TriSP innovatively merges weight magnitude, adjusted by activation norm, with first-order gradient sensitivity through a geometric mean, yielding a channel-level score. This method thoroughly examines the interplay between importance criteria and strategies for recovery post-pruning. The full paper is accessible on arXiv.
Key facts
- TriSP stands for Tri-Signal Structured Pruning.
- It targets large language models (LLMs).
- Structured pruning removes attention heads and MLP neurons.
- Existing methods are either gradient-based (memory-prohibitive) or activation-based (indirect).
- TriSP uses weight magnitude, activation norm, and gradient sensitivity.
- The metric uses a geometric mean to combine signals.
- The paper studies importance criterion and recovery strategy interaction.
- Published on arXiv with ID 2607.22587.
Entities
Institutions
- arXiv