Gated DeltaNet-2 Decouples Erase and Write in Linear Attention

A recent publication on arXiv presents Gated DeltaNet-2, which features a linear attention mechanism that distinguishes between erasing and writing functions through the use of channel-wise gates. This linear attention substitutes the limitless cache found in softmax attention with a fixed-size recurrent state, enabling sequence mixing in linear time and constant memory for decoding. The difficulty lies in modifying this compact memory without disrupting existing connections. Delta-rule models involve subtracting the current read prior to writing a new value, while Kimi Delta Attention (KDA) enhances forgetting through channel-wise decay. Nonetheless, the active edit still relies on a single scalar gate for both operations. Gated DeltaNet-2 improves upon Gated DeltaNet and KDA by separating these functions with distinct channel-wise erase gate b_t and write gate.