LLMs Learn In-Context via Dual Graph Encoding, Not Just Pattern Matching

A recent paper on arXiv disputes existing theories regarding the learning processes of large language models (LLMs) in context. The research employs a simplified random-walk task involving two contrasting graph structures to investigate whether LLMs depend on matching recent tokens or deducing underlying structures. The authors provide causal evidence indicating that neither explanation is sufficient on its own. Analysis through PCA reconstruction reveals that at certain mixture ratios, both graph structures are represented in orthogonal principal subspaces simultaneously, contradicting the idea of mere local transition copying. Additional methods, such as residual-stream activation patching and graph-difference steering, show that late-layer patching nearly completely transfers the preference for the clean graph, while linear steering shifts predictions as intended but fails under norm-match conditions. These results imply that LLMs perform in-context graph learning by monitoring global topology along with local transitions.