AttCal: Self-Supervised Attention Calibration for Transformer PLMs
A recent study published on arXiv (2512.20661) presents AttCal, a novel framework for attention calibration that operates without annotations and is self-supervised, specifically designed for Transformer-based pre-trained language models (PLMs). This approach tackles issues of attention dilution and attention sink effects, which lead models to concentrate excessively on irrelevant tokens. AttCal interprets attention distributions as stochastic policies to facilitate token deletion perturbations, assesses context-dependent token significance through semantic changes, and refines attention parameters using policy gradient methods. The backbone of the PLM remains unchanged, with only the attention projection layers being updated for training efficiency. Evaluations across five datasets indicate marked enhancements compared to standard PLMs. The authors, not specified in the abstract, announced the framework as a replace type on arXiv, aiming to boost text classification accuracy without the need for expensive token-level human annotations or static heuristics.
Key facts
- Paper arXiv:2512.20661v2, announced as replace type.
- Proposes AttCal, a self-supervised, annotation-free attention calibration framework.
- Addresses attention dilution and attention sink effects in Transformer PLMs.
- Uses task-space sensitivity feedback via stochastic policies and token deletion perturbations.
- Optimizes attention parameters via policy gradient.
- Freezes PLM backbone, updates only attention projection layers.
- Benchmarked on five datasets, outperforming vanilla PLMs.
- Published on arXiv, URL: https://arxiv.org/abs/2512.20661
Entities
Institutions
- arXiv