New AI Research Identifies Detrimental LoRA Modules, Proposes Evolutionary Pruning Method

A new research paper introduces Evolutionary Negative Module Pruning (ENMP), a method designed to improve the merging of multiple Low-Rank Adaptation (LoRA) experts for efficient multi-task AI deployment. The study, published on arXiv under identifier 2604.17753v1, challenges existing merging paradigms by identifying specific LoRA layers—termed negative modules—that degrade overall performance when combined. Current approaches, which rely on weight interpolation or subspace alignment, operate under the assumption that all LoRA matrices contribute positively to the merged model. ENMP employs an evolutionary search strategy to navigate the discrete, non-differentiable landscape of module selection, locating and excluding these detrimental modules before merging. This plug-and-play pruning method aims to overcome a critical bottleneck in LoRA merging, offering a more effective way to integrate multiple task-specific adaptations into a single backbone model. The research highlights a significant limitation in current techniques and provides a novel solution to enhance the efficiency and performance of multi-task AI systems.