ClientMorpher: Routing-Aware Federated Fine-Tuning for Heterogeneous LLMs
A new framework, ClientMorpher, has been proposed to address challenges in federated instruction fine-tuning of Mixture-of-Experts (MoE) large language models (LLMs). The method leverages routing signatures from pre-trained MoE models to organize client collaboration, aiming to mitigate negative transfer caused by heterogeneous instruction distributions. The paper, available on arXiv (2608.15311), highlights that existing federated MoE methods focus on parameter aggregation and personalization, overlooking routing behavior as a source of information for client collaboration. ClientMorpher is designed as a routing-aware, personalized federated instruction fine-tuning framework, which identifies which clients should collaborate during federated optimization. This approach is particularly relevant for privacy-sensitive decentralized data scenarios, where data sharing is not feasible. The framework is expected to improve the efficiency and effectiveness of federated learning with MoE LLMs, reducing computation and communication costs while scaling model capacity.
Key facts
- ClientMorpher is a routing-aware, personalized federated instruction fine-tuning framework.
- It leverages routing signatures from pre-trained MoE models to organize client collaboration.
- The framework addresses negative transfer under heterogeneous instruction distributions.
- Existing federated MoE methods overlook routing behavior for client collaboration.
- The paper is available on arXiv with ID 2608.15311.
- Federated instruction fine-tuning enables LLMs to adapt to decentralized, privacy-sensitive data without data sharing.
- MoE LLMs are attractive for federated learning due to sparse activation reducing computation and communication.
- The framework aims to identify which clients should collaborate during federated optimization.
Entities
—