MatryoshkaLoRA: Hierarchical Low-Rank LLM Fine-Tuning
A new method called MatryoshkaLoRA improves parameter-efficient fine-tuning of large language models by learning hierarchical low-rank representations. Existing LoRA techniques require a predefined static rank, necessitating exhaustive grid searches to balance efficiency and performance. Rank-adaptive solutions like DyLoRA sample ranks during training but yield suboptimal results at higher ranks due to inconsistent gradient signals across the rank hierarchy. MatryoshkaLoRA inserts a fixed diagonal matrix between LoRA adapters to enable accurate hierarchical learning, addressing data inefficiency. The approach is inspired by Matryoshka dolls, nesting representations of varying ranks. The paper is published on arXiv under identifier 2605.07850.
Key facts
- MatryoshkaLoRA is a training framework for Low-Rank Adaptation (LoRA).
- It learns hierarchical low-rank representations for LLM fine-tuning.
- Existing LoRA requires a predefined static rank and exhaustive grid searches.
- DyLoRA samples ranks during training but is suboptimal at higher ranks.
- MatryoshkaLoRA inserts a fixed diagonal matrix P between LoRA adapters.
- The method is inspired by Matryoshka dolls.
- The paper is on arXiv:2605.07850.
- It addresses data inefficiency in rank-adaptive methods.
Entities
Institutions
- arXiv