Study Finds LLMs Lag Humans in Integrating Long-Form Novel Information

A new arXiv paper, 'Attention Flows: Tracing LLM Conceptual Engagement via Story Summaries,' examines whether large language models mirror human patterns of narrative importance. Researchers aligned sentences from 150 human-written novel summaries with the chapters they reference, demonstrating the difficulty of the task. They then generated and aligned additional summaries from nine state-of-the-art LLMs. Comparing human and model-authored texts, they found stylistic differences, suggesting LLMs may not integrate information across long-form texts as effectively as humans. The findings highlight limitations in LLM comprehension despite growing context lengths, underscoring the complexity of summarization as a benchmark for conceptual understanding.