Exact Linear Attention: A New Transformer Mechanism

A recent study introduces Exact Linear Attention (ELA), a novel method that achieves linear computational efficiency for Transformer attention by leveraging the exact decomposition of kernel functions, which removes approximation errors. This innovation addresses problems like gradient explosion and token attention dilution seen in earlier linear attention methods by enforcing kernel constraints that ensure non-negativity, discriminability, and clear geometric meaning. The new kernels include the Hadamard Exp Kernel, the Summation Squared Euclidean Distance Kernel, and the Subtraction Squared Euclidean Distance Kernel. Furthermore, improvements feature a Hyper Link structure that substitutes traditional residual connections to mitigate gradient degradation, along with a Memory Lobe module that uses bidirectional linear attention to track transformation flow across layers.