Adaptive Memory Decay for Log-Linear Attention

Researchers propose learning the memory decay parameter in log-linear attention models directly from input data, replacing the fixed, content-independent parameter. The approach uses a lightweight two-layer MLP with softplus activation to produce per-token, per-level decay, allowing each Fenwick tree hierarchy level to scale independently. This addresses the rigidity of fixed decay in log-linear attention, which previously assigned uniform weights across hierarchy levels regardless of content. The method aims to improve the tradeoff between memory capacity and computational efficiency in sequence models, building on the log-linear attention architecture that organizes memory across a Fenwick tree hierarchy with log-linear compute cost.