Padded Transformer Expressivity: Key Architectural Factors Identified
A recent study published on arXiv examines the expressiveness of padded transformers, revealing that these models exhibit notable resilience to variations in attention mechanisms, model width, and uniformity under realistic conditions. The researchers pinpoint numeric precision and model depth as critical determinants of expressivity. They demonstrate that polynomially padded L-uniform constant-precision transformers are equivalent to L-uniform AC⁰. This research fills in the gaps in current understandings of transformer computational capabilities, utilizing padded transformers to draw connections to different circuit classes.
Key facts
- Paper on arXiv revisits padded transformer expressivity
- Padded transformers are robust to changes in attention type, model width, and uniformity
- Numeric precision and model depth are main factors affecting expressivity
- Polynomially padded L-uniform constant-precision transformers equivalent to L-uniform AC⁰
- Research addresses gaps in existing characterizations of transformer computation
- Padded transformers serve as gadget for circuit class equivalences
Entities
Institutions
- arXiv