Study Reveals How LLMs Read Bug Reports for Automated Program Repair
An empirical study has been conducted to explore attention distribution in Large Language Models (LLMs) utilized for automated program repair. The research examines 319 actual bugs in Python and Java sourced from SWE-bench Verified and Multi-SWE-bench, aiming to reveal how LLMs prioritize details in bug reports. It focuses on two key questions: (RQ1) the allocation of model attention across various sections of bug reports, and (RQ2) the differences in attention patterns between successful and unsuccessful repairs. The results offer valuable insights into why LLMs can produce accurate patches for certain bugs while struggling with similar ones, despite having the same contextual information. This study is pioneering in analyzing attention mechanisms in LLM-driven program repair, enhancing the understanding of model behavior and potential strategies to boost repair success rates.
Key facts
- First empirical study of attention patterns in LLM-based program repair
- Analyzes 319 real-world Python and Java bugs
- Uses SWE-bench Verified and Multi-SWE-bench datasets
- Investigates how model attention is distributed across bug report sections
- Compares attention patterns between successful and unsuccessful repairs
- Aims to explain inconsistent performance of LLMs in automated program repair
- Provides interpretable insights into how LLMs process bug reports
- Published on arXiv with ID 2607.25873
Entities
Institutions
- arXiv