Forward Pass Domain Adaptation: FPO Achieves 2.7-3.2x Throughput in LLM Fine-Tuning

A recent study published on arXiv (2608.14563) presents Forward-Pass-Only MLP training (FPO), a technique that modifies large language models without requiring a backward pass. FPO offers a throughput increase of 2.7–3.2 times compared to traditional fine-tuning, while consuming around 40% less peak training memory. It maintains off-domain benchmarks within seed-noise of the baseline, a consistency not typically achieved by full-network fine-tuning. This approach stems from the finding that in the later layers of a transformer, the output-layer prediction error closely resembles the true gradient, showing cosine similarities between 0.47 and 0.59 across six public models. The authors provide a two-minute diagnostic tool to assess this approximation for any model, pinpointing where late-layer adaptation is feasible. FPO generates a single error signal at the output for application to each target layer, without propagating signals or constructing an autograd graph. The paper assesses FPO across three model families and is categorized as a 'cross' announcement, indicating it is a cross-posted preprint. The implications of this method are significant, potentially lowering computational demands and memory usage in AI research and application.