FFNet: MetaMixer-based Efficient Convolutional Mixer Design

A new paper on arXiv (2406.02021v3) proposes FFNet, a convolutional mixer architecture that reinterprets the Feed-Forward Network (FFN) as a memory network akin to self-attention's query-key-value mechanism. The authors argue that the FFN, often overlooked, functions as a key-value memory where input acts as query and projection weights as keys and values. Based on this, they convert self-attention into a more FFN-like efficient token mixer using only convolutions while retaining the query-key-value framework. The work aims to demonstrate that the query-key-value structure itself is crucial for competitive performance, not necessarily self-attention. The paper is a cross-replacement announcement on arXiv.