Graph Signal Processing Reveals How LLMs Organize Numerical Sequences
A recent preprint available on arXiv (2608.03015) explores the internal representation of numerical sequences by large language models (LLMs) during in-context learning (ICL). The researchers utilize a graph signal processing framework, where attention mechanisms create weighted graphs from tokens, and hidden states act as signals on the graph nodes. Their analysis, which includes quantitative graph-spectral diagnostics and qualitative visualizations of token graphs, reveals that increased context length leads to more distinct representations based on the complexity of input dynamics. Simpler inputs result in token graphs with enhanced global connectivity and more uniform hidden-state signals. This research goes beyond merely assessing output accuracy, providing insights into how LLMs manage structured numerical data, potentially enhancing their numerical reasoning skills.
Key facts
- Preprint arXiv:2608.03015 focuses on LLM in-context learning of numerical sequences.
- Adopts a graph signal processing perspective to study internal representations.
- Attention induces weighted graphs over tokens; hidden states are signals on nodes.
- Representations become more differentiated by input dynamical complexity with longer context.
- Simpler inputs produce token graphs with stronger global connectivity.
- Simpler inputs yield smoother, spectrally concentrated hidden-state signals.
- Study goes beyond output-level evaluations like prediction error.
- Research aims to understand internal organization of numerical information in LLMs.
Entities
Institutions
- arXiv