Grammatically-Guided Sparse Attention for Efficient Transformers

A new paper on arXiv (2605.24518) introduces Grammatically-Guided Sparse Attention, a method that uses Parts-of-Speech (POS) tags to constrain attention computations in Transformer models. This approach dynamically generates attention masks that enforce linguistically coherent connections between tokens, reducing computational complexity while preserving essential linguistic dependencies. Two masking strategies are proposed: a hard mask that strictly limits interactions to predefined grammatical roles, and a soft mask that biases attention toward those roles. The work aims to address the quadratic complexity bottleneck of self-attention in long-sequence processing and large language model deployment, building on prior sparse attention methods like DeepSeek Sparse Attention.