Dual-Flow Transformers Cut Inference Costs by Decoupling Prefill and Decode

The introduction of the Dual-Flow Transformer marks a significant advancement in architecture, distinguishing the main prompt-processing route from the auxiliary computations that are only utilized during autoregressive decoding. This innovation responds to the increasing relevance of cumulative inference costs in large language models, which now approach the expenses of one-time training. The main flow operates as a full causal language model, managing prompt prefill and maintaining a persistent key-value (KV) cache. Meanwhile, the auxiliary flow is inactive during prompt processing and engages only after the final prompt position, enhancing computation for continuation prediction without impacting initial processing. This method permits extra learned computation for decoding without raising prefill costs, unlike traditional scaling methods. The research, accessible on arXiv (2608.12385), underscores the distinct hardware requirements for prefill, which is parallel and compute-bound, versus decode, which is sequential and memory-bandwidth-bound. By separating these pathways, the Dual-Flow Transformer seeks to improve resource efficiency and lower overall inference costs, addressing the rising need for real-time responses.