Topology-Aware Data Movement for Disaggregated GPU Inference
A new paper on arXiv (2607.28633) presents a system aimed at improving data movement in disaggregated GPU inference, which is a big issue when working with large language models (LLMs). When the prefill and decode phases use different GPU pools, there's a need to transfer the KV cache, which amounts to 2.6 GB per request for a 70B model and can exceed 100 GB/s in total at production levels. Existing solutions like DistServe, Splitwise, and Mooncake rely on uniform RDMA but ignore the significant bandwidth differences based on GPU locations—ranging from 900 GB/s via NVLink to 12.5 GB/s over TCP. The new orchestrator optimizes data transport by recognizing interconnect hierarchies at startup and uses several techniques to enhance efficiency, significantly reducing latency. This research is vital for AI infrastructure and addresses a key challenge in current disaggregated inference setups.
Key facts
- Paper arXiv:2607.28633 proposes topology-aware data movement for disaggregated GPU inference.
- Disaggregated LLM inference requires transferring KV cache between prefill and decode GPU pools.
- For a 70B model, KV cache transfer is 2.6 GB per request, exceeding 100 GB/s aggregate at production scale.
- Existing systems (DistServe, Splitwise, Mooncake) use uniform RDMA, ignoring bandwidth variations.
- Bandwidth varies by 72x: 900 GB/s via NVLink within a domain, 50 GB/s via InfiniBand across nodes, 12.5 GB/s via TCP across data centers.
- The orchestrator discovers interconnect hierarchy at startup and selects optimal transport per transfer.
- Pipelined layer-by-layer transfer hides 60-85% of latency behind computation.
- NVLink domain-aware placement is used for Mixture-of-Experts mode.
- The paper is a cross-type announcement on arXiv.
- The work addresses a critical networking problem in disaggregated inference architectures.
Entities
Institutions
- arXiv