GLIDE: Efficient LLM Inference via Guided Layerwise Hybrid Attention
Researchers propose GLIDE, a method to reduce memory I/O and computational overhead in Large Language Models during decoding. GLIDE integrates sliding-window softmax attention with linear recurrent aggregation, exploiting layer-wise heterogeneity: early layers are sensitive to softmax removal, while deeper layers tolerate linear alternatives. A layer-wise adaptive mechanism balances efficient linear recurrence with variable-sized softmax windows, non-uniformly compressing the softmax footprint. Empirical evaluation shows reduced KV cache I/O while preserving expressive power. The paper is available on arXiv.
Key facts
- GLIDE stands for Guided Layerwise Hybrid Attention.
- It addresses memory I/O and computational overhead of KV cache during LLM decoding.
- Early layers exhibit high sensitivity to softmax removal.
- Deeper layers demonstrate redundancy and tolerate linear alternatives.
- GLIDE uses a layer-wise adaptive mechanism.
- It integrates sliding-window softmax attention with linear recurrent aggregation.
- Non-uniform compression of softmax footprint reduces aggregate KV cache I/O.
- Empirical evaluation shows preserved expressive power.
Entities
Institutions
- arXiv