WaveFT: Wavelet-Based Sparse Fine-Tuning for Efficient Adaptation of Large Pretrained Models
A recent study introduces Wavelet Fine-Tuning (WaveFT), a technique designed for the effective adaptation of large pretrained models while adhering to strict compute and memory constraints. WaveFT focuses on learning sparse updates within the wavelet domain of weight matrices, allowing for precise management of trainable parameters below LoRA's minimum rank. Utilizing wavelet bases, which create semi-local receptive fields, this method captures spatially coherent gradients more effectively than direct weight sparsity (SHiRA) and circumvents the issues associated with global Fourier bases (FourierFT). Theoretical analysis in the paper indicates that sparse methods can produce high-rank updates, overcoming LoRA's subspace limitations and enhancing representational capacity. The research is accessible on arXiv under identifier 2505.12532.
Key facts
- Paper title: Exploring Sparsity for Parameter Efficient Fine Tuning Using Wavelets for Vision
- arXiv ID: 2505.12532
- Proposes Wavelet Fine-Tuning (WaveFT) for parameter-efficient fine-tuning
- WaveFT learns sparse updates in the wavelet domain of weight matrices
- Enables fine-grained control over trainable parameters below LoRA's minimum rank
- Wavelet bases provide semi-local receptive fields that aggregate spatially coherent gradients
- Theoretical analysis shows sparse methods achieve high-rank updates, avoiding LoRA's subspace bottleneck
- Paper is a preprint with announcement type 'replace-cross'
Entities
Institutions
- arXiv