Ortho-Hydra: Solving Style Bleed in DiT LoRA Fine-Tuning
A novel approach known as Ortho-Hydra tackles the issue of style bleed encountered in the LoRA fine-tuning of diffusion transformers (DiT) when dealing with multi-style datasets. Style bleed arises when a single low-rank residual fails to encapsulate various distinct artist signatures, leading the optimizer to settle on their average. In the HydraLoRA framework, the mixture-of-experts LoRA substitutes the up-projection with E heads managed by a router, but the zero initialization causes identical gradients and uniform routing, which results in permutation-symmetric experts training as a single rank-r LoRA at an E× expense. Ortho-Hydra innovatively reparameterizes this by integrating an OFT-style Cayley-orthogonal shared basis with separate output subspaces for each expert, derived from the top-(Er) left singular vectors of the pretrained weights. This disjointness guarantees that the router's per-expert score is non-degenerate from the outset, facilitating specialization immediately. The method is detailed in a paper available on arXiv under ID 2605.03252.
Key facts
- Ortho-Hydra is a reparameterization for LoRA fine-tuning of diffusion transformers.
- It addresses style bleed in multi-style data fine-tuning.
- Style bleed occurs when a single low-rank residual averages multiple artist styles.
- HydraLoRA style uses E heads under a router but suffers from uniform routing due to zero initialization.
- Ortho-Hydra uses a Cayley-orthogonal shared basis and per-expert disjoint output subspaces.
- Disjoint subspaces are carved from top-(Er) left singular vectors of pretrained weights.
- The method ensures non-degenerate router scores at initialization.
- The paper is available on arXiv with ID 2605.03252.
Entities
Institutions
- arXiv