CoRe: Shifting Continual Learning Finetuning from Weight Space to Representation Space

A recent paper on arXiv presents Continual Representation Learning (CoRe), a novel framework that transforms the concept of finetuning for continual learning by focusing on representation space instead of weight space. Cited as arXiv:2603.11201v3, the research critiques existing Parameter-Efficient Fine-Tuning (PEFT) techniques, which depend on opaque, empirical optimization at the weight level, resulting in a lack of control over representation drift. This shortcoming contributes to vulnerability to domain changes and catastrophic forgetting in continual learning contexts. CoRe mitigates these issues by implementing task-specific adjustments within a low-rank linear subspace of hidden representations, utilizing a learning process with clear objectives. This shift from weight-level to representation-level finetuning could enhance adaptability to changing data streams. The full paper can be found at https://arxiv.org/abs/2603.11201.