BaLoRA: Bayesian Low-Rank Adaptation Improves Fine-Tuning Accuracy and Uncertainty
Researchers have introduced BaLoRA, a Bayesian extension of Low-Rank Adaptation (LoRA) for fine-tuning large pre-trained models. LoRA reduces computational cost but suffers from limited expressiveness and lacks uncertainty quantification. BaLoRA adds minimal parameters and compute through an input-adaptive Bayesian parameterization of LoRA matrices. The method yields well-calibrated uncertainty estimates and, surprisingly, improves prediction accuracy via adaptive noise injection, narrowing the gap with full fine-tuning on natural language reasoning and vision tasks. In band gap prediction for metal-organic frameworks, BaLoRA achieves zero-shot test-time performance. The work addresses a key limitation of LoRA in reliability-critical applications.
Key facts
- BaLoRA is a Bayesian extension of Low-Rank Adaptation (LoRA).
- LoRA is standard for fine-tuning large pre-trained models at reduced cost.
- LoRA's low-rank point-estimate updates limit expressiveness and accuracy.
- LoRA provides no built-in uncertainty quantification.
- BaLoRA uses an input-adaptive Bayesian parameterization of LoRA matrices.
- BaLoRA adds minimal parameters and compute.
- BaLoRA yields well-calibrated uncertainty estimates.
- Adaptive noise injection in BaLoRA improves prediction accuracy.
- BaLoRA narrows the gap with full fine-tuning on natural language reasoning and vision tasks.
- BaLoRA was applied to band gap prediction in metal-organic frameworks.
- BaLoRA produces zero-shot test-time performance for band gap prediction.
Entities
Institutions
- arXiv