Research Reveals How to Teach AI Models New Skills Without Forgetting Old Ones

A recent study explores methods for teaching large multimodal models new abilities while preserving their existing knowledge. Researchers focused on sequential fine-tuning across five specific skills and monitored overall performance across eight different benchmarks. The investigation involved three unique model families. Surprisingly, it was discovered that performance on previously held-out tasks could partially recover after fine-tuning on a different skill, linked to a notable change in the model's output token distribution. A straightforward counting-bias probe indicated that this change correlates with forgetting. The study proposed two effective tuning strategies that enhance learning while minimizing detrimental drift. The first method adjusts only the self-attention projection layers, yielding a +24.9 learning gain and a -0.6 reduction in held-out forgetting. The second method modifies only the MLP Gate and Up projections while keeping the Down projection static, achieving +30.5 and -2.1 gains respectively. Both strategies significantly surpassed full fine-tuning. This research was published on arXiv with the identifier 2510.08564v2, categorized as a 'replace' announcement.