MIITA: Memory-Based Continual Learning for Small Language Models
Researchers propose MIITA, a Memory-Induced Inference-Time Adaptation framework for continual learning with small language models (SLMs). The method addresses catastrophic forgetting by storing supervised experiences as compact correction-direction prototypes with semantic anchors, retrieved at inference using semantic and uncertainty cues. Gated temporary hidden-state adaptation enables non-destructive reuse of past supervision without updating model parameters. The approach targets resource-constrained deployments where SLMs must adapt to evolving real-world needs.
Key facts
- MIITA stands for Memory-Induced Inference-Time Adaptation
- Designed for supervised continual learning under constrained storage
- Stores experiences as correction-direction prototypes with semantic anchors
- Retrieval uses semantic and uncertainty-based cues
- Applies gated temporary hidden-state adaptation
- Avoids catastrophic forgetting without backbone updates
- Targets small language models in resource-constrained deployments
- Published on arXiv with ID 2607.22556
Entities
Institutions
- arXiv