FedPA-LoRA: New Framework for Heterogeneous Federated LoRA
A recent study presents FedPA-LoRA, a framework aimed at reducing aggregation and initialization inaccuracies within heterogeneous federated Low-Rank Adaptation (LoRA). This research, accessible on arXiv with the identifier 2608.15381, tackles the challenge of achieving precise aggregation of local updates while ensuring the continuity of locally optimized factors during the federated fine-tuning of large language models. While factor-wise aggregation maintains factor continuity, it leads to aggregation mismatches. Conversely, product-space reconstruction minimizes mismatch but increases initialization discrepancies at the factor level. FedPA-LoRA effectively resolves these issues and guarantees convergence for both homogeneous and heterogeneous client ranks. Clients maintain their local factors throughout communication rounds and align their products with a rank-specific global reference, ensuring local optimization continuity and global consistency amidst data diversity. The server handles the aggregation of updates from varying ranks.
Key facts
- Paper introduces FedPA-LoRA, a product-aligned federated LoRA framework.
- Addresses aggregation and initialization errors in heterogeneous federated LoRA.
- Proposes method to mitigate factor-wise aggregation mismatch and initialization mismatch.
- Provably converges under homogeneous and heterogeneous client ranks.
- Each client preserves local factors and aligns product to rank-specific global reference.
- Maintains local optimization continuity and global consistency under data heterogeneity.
- Server aggregates heterogeneous-rank updates.
- Available on arXiv with ID 2608.15381.
Entities
Institutions
- arXiv