Adjoint Matching Framework for Fine-Tuning Flow Models

A novel deterministic adjoint matching framework approaches the alignment of human preferences in flow-based generative models by treating it as an optimal control problem related to velocity fields. This technique directly regresses control towards a target influenced by value gradients under the existing policy, resulting in a straightforward and stable training goal. By employing a truncated adjoint method, the focus is placed on the latter part of the trajectory, where signals relevant to rewards are concentrated, leading to significant computational efficiency while maintaining alignment quality. This framework extends beyond typical KL-based regularization, enabling adaptable trade-offs between the strength of alignment and the preservation of distributions. Experiments conducted on SiT-XL/2 and FLUX.2-Klein-4B reveal consistent improvements across various alignment metrics, as well as enhanced diversity and mode preservation.