Attention-Guided Layer Selection Boosts LLM Factuality

Researchers propose three attention-guided strategies—Attention-JSD, Attention-Entropy-Max, and Attention-Entropy-Min—to improve contrastive decoding in large language models. These methods leverage self-attention distributions as a signal for layer selection, outperforming the original DoLa approach on the TruthfulQA benchmark. Significant gains are observed on multi-answer metrics MC2 and MC3, indicating that attention distributions provide a more sensitive signal for factual knowledge retrieval than output vocabulary distributions alone.