Bayesian Model Merging: A Bi-Level Optimization Framework

A recent publication on arXiv presents Bayesian Model Merging (BMM), a bi-level optimization strategy designed to integrate various task-specific expert models into one cohesive model without the need for joint retraining. The inner level approaches merging through activation-based Bayesian regression, leveraging a robust prior from an anchor model, which results in a closed-form solution. Meanwhile, the outer level employs Bayesian optimization to globally explore hyperparameters specific to each module. This framework effectively tackles two significant drawbacks of current techniques: the neglect of inductive bias from anchor models and the dependence on uniform hyperparameters across different network modules.