Meta-Adaptive Network Gradient Optimization for Online Continual Learning

A new method called Meta-Adaptive Network Gradient Optimization (MANGO) is proposed to address catastrophic forgetting in Online Continual Learning (OCL). OCL involves a neural network learning sequentially from a non-stationary data stream in a single pass with a limited memory replay buffer, unlike offline continual learning which relies on multiple epochs over large datasets. The key challenge is balancing stability (retaining past tasks) and plasticity (learning new tasks). Existing approaches such as replay-based rehearsal, output-level distillation, fixed regularization, and stream-only meta-learning have limitations: rehearsal introduces stored sample bias; distillation does not modulate parameter updates; fixed regularization penalizes parameters irrespective of sensitivity; stream-only meta-learning lacks feedback-controlled parameter updates. MANGO aims to overcome these issues by introducing a meta-adaptive gradient optimization framework that dynamically adjusts parameter updates based on task relevance and sensitivity. The method is detailed in a preprint on arXiv (2605.19080).