Head-Wise Adaptive Sparse Attention Accelerates Video Diffusion

A new training-free method, HASTE, accelerates video diffusion models by addressing inefficiencies in sparse attention. Current training-free sparse attention suffers from high mask prediction costs and uniform thresholds across attention heads, limiting speed-quality trade-offs. HASTE introduces two components: Temporal Mask Reuse, which skips unnecessary mask prediction by tracking query-key drift, and Error-guided Budgeted Calibration, which assigns per-head sparsity thresholds to minimize model-output error under a global budget. Tested on Wan2.1-1.3B, the method improves efficiency without retraining.