New Benchmark and Method for Continual Model Routing in AI Hubs

To tackle the issue of model selection and updates in swiftly evolving AI model hubs, researchers have established Continual Model Routing (CMR). They have also launched CMRBench, an extensive benchmark that replicates realistic hub growth with more than 2,000 potential models. Additionally, the team has introduced CARvE, a contrastive embedding technique that employs checkpoint-based anchoring and structured replay to enhance routing efficiency. Empirical findings indicate that CARvE surpasses zero-shot retrieval, fine-tuning, and adapter-merging benchmarks in terms of accuracy at the model, family, and domain levels.