Transformers' Expressive Power Mapped via Circuit Complexity
A new paper on arXiv (2608.12671) surveys how multi-layer transformers, the core of large language models, compare to classical computational models. The study uses circuit complexity to analyze transformer capabilities, parameterizing them by resources like attention and precision to draw parallels with circuit classes. The overview highlights key results that define the boundaries of what transformers can recognize as language processors.
Key facts
- The paper is available on arXiv with ID 2608.12671.
- It focuses on multi-layer transformers, which are critical components of LLMs.
- The research compares transformers to standard models of computation.
- Circuit complexity is identified as the appropriate framework for analysis.
- Transformers are parameterized by resources such as attention and precision.
- The comparison involves circuit classes parameterized by gate type, size, and depth.
- The paper presents an overview of selected results on transformer expressive power.
- The announcement type is 'new'.
Entities
Institutions
- arXiv