Retraction-Free Optimization for LoRA Fine-Tuning on Stiefel Manifold
A novel algorithm that operates without retractions or penalty parameters for optimization on the Stiefel manifold has been introduced, specifically aimed at fine-tuning large language models using LoRA. This approach eliminates the need for expensive orthonormalization and meticulous step size adjustments by directly operating on the manifold. It utilizes the strongly-convex-like characteristics of the quadratic penalty function alongside the proximal smoothness of the Stiefel manifold, ensuring global convergence with optimal iteration complexities for both constant and diminishing step sizes. The authors recast the LoRA fine-tuning challenge as a manifold optimization issue, presenting Manifold-LoRA for enhanced geometric adaptation. This research is documented in arXiv:2607.25299.
Key facts
- Proposed algorithm is retraction-free and penalty parameter-free.
- Avoids costly orthonormalization for large-scale matrices.
- Leverages strongly-convex-like property of quadratic penalty function.
- Uses proximal smoothness of Stiefel manifold.
- Achieves global convergence with best-known iteration complexities.
- Reformulates LoRA fine-tuning as manifold optimization problem.
- Introduces Manifold-LoRA for geometry-accelerated adaptation.
- Published on arXiv with ID 2607.25299.
Entities
Institutions
- arXiv