LoSA: Near-Lossless Sparse Attention for Training-Free Video Diffusion Acceleration
A new method called LoSA, or Near-Lossless Sparse Attention, has been developed to improve how video diffusion transformers work, and it doesn't require any retraining. This technique tackles the issue of high costs associated with self-attention in lengthy 3D token sequences, which can become more problematic with higher resolutions and longer durations. Unlike existing sparse attention methods that focus on extreme sparsity, LoSA keeps a threshold of 99% of the attention mass, ensuring nearly perfect fidelity. It’s based on two main ideas: around 40% of block interactions can be cut without losing attention mass, and the high-mass support remains stable during denoising. You can check out the research on arXiv with ID 2608.12032.
Key facts
- LoSA is a training-free sparse-attention method for video diffusion transformers.
- It fixes a retained-mass threshold of 99% instead of a sparsity ratio.
- About 40% of block interactions can be removed while retaining 99% of attention mass.
- The high-mass support remains stable across denoising steps.
- LoSA measures exact block attention masses at one early dense step.
- The method targets the quadratic cost of self-attention over long 3D token sequences.
- The paper is available on arXiv with ID 2608.12032.
- The announcement type is cross.
Entities
Institutions
- arXiv