Progressive Multimodal Alignment for Continual Instruction Tuning

So, there's this new framework called Progressive Multimodal Alignment (PMA) that addresses a problem known as projector-level forgetting in Multimodal Continual Instruction Tuning (MCIT). In Multimodal Large Language Models (MLLMs), a projector connects visual data to language. However, as visual data changes and instruction meanings shift, the projector can drift. PMA detects these changes using a simple representation descriptor and only adds new projector experts when needed. It also uses an expandable router to mix expert outputs based on multimodal features, while keeping the original pretrained projector as a stable reference. This approach helps adapt the projector continuously without losing earlier alignments, fixing a gap in existing methods that focus mainly on the LLM backbone.