Research Reveals Optimizer Dynamics Shape AI Model Merging Effectiveness

There’s a new study that looks at how optimization dynamics affect the shape of loss landscapes when AI models are combined, which plays a big role in how well different solutions can be integrated. You can find the paper on arXiv under the ID arXiv:2510.04686v2. It examines two common methods: linear interpolation, which blends model weights, and task arithmetic, which combines task vectors by looking at differences between finetuned and base models. The research identifies a key metric called effective noise scale that captures how various optimizer elements impact merging. It reveals a complex link between merging success and this noise scale, influenced by factors like learning rates and data augmentation. The study highlights that while merging models can enhance capabilities without increasing costs, the underlying principles are still not fully clear.