LoRA-Over: Enhancing LLM Adaptation with Strategic Over-Parameterization
A new framework called LoRA-Over improves the generalization of Low-Rank Adaptation (LoRA) for large language models (LLMs) by strategically over-parameterizing during training and collapsing the enrichment at inference. The method injects auxiliary parameters into low-rank adapters to broaden the hypothesis space, then folds them back via decomposition-based reformulation. This addresses the tension between parameter efficiency and adaptation capacity, achieving better transfer across heterogeneous tasks and domains without increasing inference cost. The approach is detailed in arXiv:2605.16470, submitted on May 28, 2025.
Key facts
- LoRA-Over is a framework for parameter-efficient fine-tuning of LLMs.
- It injects auxiliary parameters during training to enrich the optimization landscape.
- The enrichment is collapsed at inference via decomposition-based reformulation.
- It aims to improve generalization across heterogeneous tasks and domains.
- The method does not increase inference cost.
- The paper is available on arXiv with ID 2605.16470.
- The approach revisits the trade-off between parameter efficiency and adaptation capacity.
- LoRA-Over builds on Low-Rank Adaptation (LoRA).
Entities
Institutions
- arXiv