LoRA Fails to Learn Multi-Step Procedures in LLMs
A recent study published on arXiv highlights limitations in fine-tuning AI models, particularly with procedural knowledge that involves multiple steps. In a travel booking analysis involving 14 nodes, models using LoRA with ranks from 16 to 128 recorded a success rate of just 2.54, compared to 4.11 for complete fine-tuning, with a significant statistical difference (p<0.001). Both techniques maintained high conversation completion rates of 95-99%. Further evaluations in different scenarios, including a Zoom support setup with 14 nodes and a 55-node insurance claims process, revealed LoRA consistently underperformed by 0.8 to 2.2 points against full fine-tuning, especially in complex tasks.
Key facts
- LoRA fails to match full fine-tuning on procedural tasks with conditional branching.
- Study used travel booking (14 nodes), Zoom support (14 nodes), and insurance claims (55 nodes).
- Task success for LoRA was ≤2.54 vs. 4.11 for full fine-tuning on travel booking.
- All LoRA configurations (r=16–128) performed uniformly poorly.
- Conversation completion rates remained 95–99% despite procedural failure.
- Cross-domain replication at 8B scale confirmed the pattern.
- LoRA underperformed by 0.8–2.2 points on average at r=32 and r=128.
- The largest performance gap was in the most complex tasks.
Entities
Institutions
- arXiv