Echo-LoRA: Cross-Layer Injection for Efficient LLM Fine-Tuning

Researchers propose Echo-LoRA, a parameter-efficient fine-tuning method that injects cross-layer representations into shallow LoRA modules. During training, boundary hidden states from deeper source layers are aggregated into a sample-level echo representation, then projected via lightweight networks into shallow adapters. Techniques like answer-only masking, masked distillation, and stochastic routing stabilize the auxiliary path and reduce train-inference gap. Evaluated on eight commonsense reasoning benchmarks, Echo-LoRA outperforms standard LoRA and DoRA baselines while adding minimal parameters. The method is published on arXiv under ID 2605.08177.