RMM: A Training-Free Policy for Fixed-Lag Smoothing in LLM Memory Management

A recent paper on arXiv (2607.24667) reinterprets the challenge of handling a language model's limited working memory as an estimation issue regarding a hidden signal—specifically, the likelihood of an item being reused. Current approaches, including StreamingLLM, H2O, and SnapKV, implement eviction as soon as an item is received (with commit lag H=0), whereas Belady's optimal offline strategy necessitates complete foresight. The authors introduce a novel approach termed fixed-lag smoothing, which observes a limited number of steps to determine which items a near-future prediction focused on before committing. This technique transforms Belady's unobservable future requests into accessible data from the model. They present this as RMM, a training-free policy that generalizes H2O, reverting to it when the measurement is uniform. The paper contributes theoretically to efficient memory management in LLMs, providing fresh insights into eviction tactics.