Toeplitz MLP Mixer: Efficient Sequence Model with Linear Complexity

A new architecture called the Toeplitz MLP Mixer (TMM) has been developed by researchers, which substitutes traditional attention mechanisms with triangular-masked Toeplitz matrix multiplication. This innovation allows for O(dn log n) time and O(dn) space complexity during training, and O(dn) for both time and space during inference prefill, surpassing the quadratic complexity associated with standard attention. Although TMMs do not incorporate advanced input modulation or state retention, they demonstrate enhanced training efficiency in terms of loss per compute and device memory. Additionally, they are capable of retaining more input information, resulting in better copying performance and improved information retrieval. The full paper can be found on arXiv.