LLM Reasoning Steps Localized in Attention Heads
A new arXiv preprint (2605.27824) investigates how Large Language Models (LLMs) perform logical reasoning by localizing attention heads responsible for individual reasoning steps. The study uses a symbolic-aided Chain-of-Thought (CoT) prompting framework to align reasoning steps with token logits. It finds that token positions steering reasoning are associated with low confidence scores due to constraints from demonstrations. The work aims to characterize information transfer among attention heads, revealing how LLMs understand abstract reasoning from limited examples.
Key facts
- arXiv preprint 2605.27824
- Announce type: new
- Focuses on LLM reasoning
- Uses symbolic-aided CoT prompting
- Localizes attention heads for reasoning steps
- Token positions with low confidence steer reasoning
- Constraints from demonstrations affect confidence
- Characterizes information transfer among attention heads
Entities
Institutions
- arXiv