SPDP: Unified Static-Dynamic Pruning for Efficient LLM Inference
A novel framework named SPDP combines unstructured static pruning with input-adaptive dynamic pruning to enhance the efficiency of large language model (LLM) inference on GPUs. This approach tackles the computational and memory limitations associated with autoregressive decoding, which is primarily affected by bandwidth-bound kernels. SPDP features a Tiled-Column-wise Bitmap Compressed (Tiled-CBC) format along with two complementary GPU kernels: a CUDA-core spMspV kernel that employs Hybrid Activation-aware Dynamic Shared-Memory Bitmap Decoding (HAD-SMBD) for precise runtime activation skipping, and a Tensor-Core SpMM kernel. The research can be accessed on arXiv using the identifier 2607.21985.
Key facts
- SPDP unifies static and dynamic pruning for LLM inference.
- It uses a Tiled-Column-wise Bitmap Compressed (Tiled-CBC) format.
- Two GPU kernels are designed: CUDA-core spMspV with HAD-SMBD and Tensor-Core SpMM.
- The method targets autoregressive decoding bottlenecks.
- Paper available on arXiv: 2607.21985.
- Weight pruning is used to reduce computational and memory costs.
- Static pruning removes redundant weights permanently.
- Dynamic pruning adapts to input sparsity but introduces irregularity.
Entities
Institutions
- arXiv