LLM Bug Report Summaries Contain 12.3% Fabricated Content

A study on 80 structured bug report summaries generated by large language models found that 47.9% contained missing information and 12.3% included fabricated content. The research, published on arXiv (2605.24137), analyzes hallucinations from a section-aware perspective, focusing on Steps-to-Reproduce, Actual Behavior, and Expected Behavior sections. Existing detection methods evaluate at the full-response level and ignore technical document structure. The findings highlight the need for systematic hallucination analysis in automated bug report summarization to prevent misleading developers.