LLMs Spontaneously Reconstruct Graph Topology Internally, Study Finds

A recent preprint on arXiv (2605.10503) indicates that Large Language Models (LLMs) inherently reconstruct graph structures while handling serialized graph information, as shown by a 'sawtooth' pattern in attention maps that corresponds with a token-level adjacency matrix. Nevertheless, this natural comprehension of structure is compromised by the attention sink, where models excessively concentrate on specific tokens. The authors define this compromise as a representation bottleneck, arising from a clash between the model's anisotropic bias—beneficial for language tasks—and the local aggregation necessary for graph reasoning. Current approaches, such as external graph-based adapters or fine-tuning, are expensive and hinder generalizability. The paper suggests enhancing structural attention within LLMs to improve graph reasoning efficiency without external tools.