First Theoretical Framework for Spiking Transformers Establishes Expressivity and Efficiency

A groundbreaking theoretical model for spiking transformers has been introduced, achieving comparable accuracy to traditional transformers while enhancing energy efficiency by 38 to 57 times on neuromorphic hardware. The study, available on arXiv with the identifier 2604.15769v1, presents the inaugural comprehensive expressivity theory for spiking self-attention. It demonstrates that spiking attention utilizing Leaky Integrate-and-Fire neurons can universally approximate continuous permutation-equivariant functions. The paper includes explicit spike circuit designs, featuring an innovative lateral inhibition network for softmax normalization that shows O(1/√T) convergence. Additionally, it establishes tight lower bounds on spike counts through rate-distortion theory, indicating that ε-approximation necessitates Ω(L_f² nd/ε²) spikes. Notably, input-dependent bounds are derived from effective dimensions, ranging from 47 to 89 for datasets such as CIFAR and ImageNet, clarifying why T=4 time steps are often sufficient despite theoretical expectations. This research fills a significant gap in the theoretical framework for creating these energy-efficient neural network models.