Linearized 2-Simplicial Attention Achieves Linear Cost with Global Reach

A recent research article presents a linearized variant of 2-simplicial attention, which enhances conventional attention mechanisms to accommodate higher-order interactions. The authors reformulate the trilinear score as an inner product involving a composite query and a key, enabling the summation across one token axis to mirror the structure of standard softmax attention. They utilize positive random features to approximate this summation, preserving the entire history in a fixed-size state while explicitly maintaining the second axis over a brief window of recent tokens. This approach achieves linear complexity concerning sequence length while offering a global perspective absent in windowed 2-simplicial attention. Implemented with custom Triton kernels and integrated with Kimi Delta Attention, this model operates without softmax attention. When matched for compute, it demonstrates the highest mean downstream accuracy compared to other architectures. At a context length of 16k, it enhances mean accuracy over a KDA hybrid and reduces LAMBADA perplexity from 715.6 to 602.6. The paper can be accessed on arXiv with the identifier 2608.09307 in the Computer Science > Artificial Intelligence section.