IFCLoRA: Topology-Aware Rank Allocation for Efficient LLM Fine-Tuning

The newly introduced technique, IFCLoRA, enhances rank distribution in Low-Rank Adaptation (LoRA) for extensive language models by examining the information flow topology prior to fine-tuning. In contrast to current adaptive-rank approaches that depend on local gradient statistics during training, IFCLoRA utilizes a small calibration dataset along with a frozen pretrained model to create a sparse interaction graph conditioned on tasks. The nodes in this graph signify LoRA-compatible modules. By merging a global information-flow topology prior with local gradient sensitivity, it calculates Information-Flow Centrality scores, assessing the adaptation significance of each module through multi-hop propagation. Ranks are allocated once within a global budget, minimizing additional memory and computation while integrating task-conditioned global insights. The research can be found on arXiv with ID 2607.22251.