Bridging Linear and Non-Linear Fine-Tuning for Task Vectors

A recent paper on arXiv (2605.18993) introduces a technique that merges the advantages of both linear and non-linear fine-tuning for composing task vectors in pre-trained models. Task vectors facilitate model integration through addition and enable unlearning through subtraction. While linear fine-tuning in tangent space yields disentangled vectors that are less prone to interference, it also restricts expressivity and raises computational demands. To address this issue, the authors impose linearity concerning weight perturbations by applying activation-space constraints during the training phase. They extract hidden representations from a curvature-regularized linearized teacher and use them to train a non-linear student through standard fine-tuning, allowing effective task arithmetic without the limitations of entirely linear models.