TokenMem: A Lightweight Memory System for Faithful Knowledge Injection in Frozen LLMs
TokenMem is an innovative memory architecture that allows the integration of external knowledge into static large language models (LLMs) without generating conflicts in knowledge. In contrast to retrieval-augmented generation (RAG), which utilizes a common self-attention pathway that may produce erratic results when retrieved data contradicts existing knowledge, TokenMem employs a separate cross-attention channel. This method avoids competing with parametric memory within the residual stream. The system trains a lightweight gating adapter, consisting of around 3-7 million parameters, through a two-phase curriculum: initially focusing on general knowledge application, followed by enhancing adherence to counterfactual information. In tests involving five models from three families (Qwen3-4B/8B/14B, LLaMA-3.1-8B, OLMo-3-7B), TokenMem demonstrated 69-70% Knowledge Compliance (KC) on counterfactual benchmarks, significantly outpacing vanilla RAG's 20-52%, with a difference of up to 49 percentage points. The research paper can be found on arXiv.
Key facts
- TokenMem is a lightweight memory system for frozen LLMs
- Uses a dedicated cross-attention channel to inject knowledge
- Trains a thin gating adapter of ~3-7M parameters
- Two-phase curriculum: general knowledge utilization then faithful compliance
- Tested on Qwen3-4B/8B/14B, LLaMA-3.1-8B, OLMo-3-7B
- Achieves 69-70% Knowledge Compliance on counterfactual benchmarks
- Vanilla RAG achieves 20-52% on same benchmarks
- Performance gap of up to 49 percentage points
Entities
Institutions
- arXiv