LoCA: Forward-Only LLM Tuning after One-Shot Calibration with Local Credit Assignment
A recent preprint on arXiv (2608.03020) presents Local Credit Assignment (LoCA), a two-step technique designed for small-shift adaptation in large language models, which removes the necessity for repeated end-to-end backpropagation through a fixed backbone. Initially, a single probe backward pass establishes a low-rank map at each transformer block, correlating the final prediction error with a local hidden-state adjustment. Subsequently, these maps are employed to create blockwise regression targets from forward activations, and low-rank adapters are adjusted using closed-form ridge solutions. This method eliminates additional backward passes through the backbone, minimizing the requirement for backward-compatible hardware and avoiding the need for activation storage or recomputation. Evaluated on five discriminative benchmarks with Qwen2.5 models ranging from 0.5B to 14B parameters, LoCA outperformed baseline methods in 16 out of 25 task-scale comparisons by achieving lower evaluation cross-entropy. The paper can be accessed on arXiv under the identifier 2608.03020.
Key facts
- LoCA is a two-stage method for small-shift adaptation of LLMs.
- It replaces repeated end-to-end backpropagation with a one-time calibration.
- A single probe backward pass fits low-rank maps at each transformer block.
- These maps are reused to form blockwise regression targets from forward activations.
- Low-rank adapters are fitted with closed-form ridge solves.
- No further backbone backward pass is required.
- Evaluated on five discriminative benchmarks with Qwen2.5 models from 0.5B to 14B.
- In 16 of 25 task-scale comparisons, LoCA yielded lower evaluation cross-entropy.
Entities
Institutions
- arXiv