F-WANDA: Efficient Pruning Method for Sustainable LLM Deployment
A recent study published on arXiv (2608.00481) presents F-WANDA, a method for post-training pruning of large language models (LLMs) that effectively balances performance and computational efficiency. In contrast to previous methods like WANDA, which compromises quality, and SPARSEGPT, known for its high computational demands, F-WANDA redistributes the per-row keep budget across output neurons utilizing the empirical Fisher information derived from pre-activation. This Fisher signal is calculated through a single additional backward pass over the same calibration dataset used by WANDA, without any weight adjustments. When tested on LLAMA-2-7B at 50% unstructured sparsity, F-WANDA achieves a WikiText-2 perplexity of 6.85, matching WANDA’s fluency, and enhances 5-shot MMLU by +1.6 percentage points over WANDA and +1.1 over SPARSEGPT, while consuming only one-third of SPARSEGPT's pruning time and energy. This is accomplished without needing extra calibration data or fine-tuning, positioning F-WANDA at the forefront of quality versus pruning cost for the sustainable deployment of LLMs.
Key facts
- F-WANDA is a drop-in modification of WANDA
- It reallocates per-row keep budget across output neurons based on empirical Fisher information
- Fisher signal collected in one additional backward pass over WANDA's calibration corpus
- No weights are updated
- On LLAMA-2-7B at 50% sparsity, WikiText-2 perplexity is 6.85
- Improves 5-shot MMLU by +1.6 pp over WANDA and +1.1 pp over SPARSEGPT
- Uses one-third of SPARSEGPT's pruning wall-clock and energy
- No extra calibration data or fine-tuning required
Entities
Institutions
- arXiv