MoNe: Modular Neural Memory Cuts Long-Context Inference Costs by 80% at 128K Tokens
MoNe presents itself as a lightweight modular neural memory for pretrained Transformers, enabling long-context inference without any retraining. Processing input in fixed chunks through fast-weight networks learned at test time, the system avoids rereading context during inference by deriving keys and values purely from query tokens. Consequently, computational requirements become independent of sequence length: preprocessing scales linearly with N, while each query costs constant time and peak GPU memory stays flat. At 128K tokens, MoNe lowers compute and memory overhead by roughly 80% relative to standard in-context learning, adding just 6.4% parameters. It also handles context windows far beyond the backbone's limit, excelling at needle-in-a-haystack and word extraction tasks from RULER, where conventional ICL fails sharply.
Key facts
- MoNe is a lightweight modular neural memory for Transformer models.
- It attaches to any frozen pretrained Transformer without retraining.
- It processes context in fixed-size segments using test-time learning of fast-weight networks.
- At inference, it generates keys and values from query tokens alone, avoiding context re-reads.
- It achieves O(N) preprocessing and O(1) query cost.
- Peak GPU memory does not grow with context length N.
- At 128K tokens, it reduces compute and peak GPU memory by approximately 80% compared to in-context learning, with only 6.4% parameter overhead.
- It generalizes beyond the backbone's native window, performing strongly on RULER benchmarks where ICL degrades sharply.
Entities
Institutions
- arXiv
- arXivLabs
- Semantic Scholar
- RULER