Spaced Repetition Training: Adaptive Replay Scheduling for Continual Pre-Training of LLMs

The research paper titled "When to Review: Spaced Repetition for Continual Pre-Training of Language Models" (arXiv: 2608.17530) introduces Spaced Repetition Training (SRT), a framework designed for continual learning in large language models. It tackles the issue of preserving previously learned knowledge while assimilating new data. Unlike traditional approaches that rely on a static blend of old and new information, SRT treats continual pre-training as an adaptive scheduling challenge, employing the SuperMemo-2 (SM-2) algorithm for organizing training example reviews. It tracks review states for each example and translates perplexity into a signal for recall quality. Assessments conducted on Wikipedia and code datasets indicate that SRT improves the stability-plasticity balance compared to standard methods, underscoring the importance of memory scheduling strategies.