PAGE Method Identifies Dominant LoRA Adapter Module
A recent research effort has unveiled PAGE (Projected Adapter Gradient Energy), a sensitivity probe based on gradients that assesses the initial trainable gradient energy for LoRA adapters. Evaluating two model families across four downstream tasks, PAGE demonstrates that gradient energy is primarily focused on a specific shallow FFN down-projection, referred to as the dominant adaptation module. While the layer index of this module is influenced by the architecture, it remains stable across tasks. Consequently, the authors introduce DomLoRA, a method for optimizing adapter placement by targeting this key module. These results challenge the prevalent approach of widely distributing adapters, indicating that strategically positioning a few can sustain or enhance performance.
Key facts
- PAGE estimates initial trainable gradient energy for LoRA adapters
- Gradient energy concentrates on a single shallow FFN down-projection
- Dominant adaptation module is architecture-dependent but task-stable
- DomLoRA is a placement method targeting the dominant module
- Study tested across two model families and four downstream tasks
- Findings suggest fewer adapters can maintain or improve performance
Entities
—