LoCO: Low-rank Compositional Rotation Fine-tuning

A new parameter-efficient fine-tuning method called LoCO (Low-rank Compositional Orthogonal fine-tuning) has been introduced. LoCO constructs orthogonal transformations via low-rank skew-symmetric matrices and compositional rotation chains, preserving the geometric structure of pretrained representations better than existing low-rank adaptations. An approximation scheme enables fully parallel computation of compositional rotations, making it practical for high-dimensional feature spaces. The method maintains low computational complexity with controlled approximation error. Validation across diverse domains includes diffusion transformers. The paper is available on arXiv under reference 2605.15916.