Queueing Theory Model for LLM Inference Stability with KV Cache Constraints
A new queueing-theoretic framework from arXiv:2605.04595 analyzes stability in large language model (LLM) inference under both computation and GPU memory constraints, specifically addressing key-value (KV) caching overhead. The study derives rigorous stability and instability conditions to determine if an LLM inference service can sustain demand without unbounded queue growth. By combining estimated request arrival rates with derived stable service rates, operators can calculate necessary cluster sizes to avoid cost overruns. This work provides a tool for GPU provisioning in LLM deployment.
Key facts
- arXiv:2605.04595 introduces the first queueing-theoretic framework for LLM inference with KV cache memory constraints.
- The framework incorporates both computation and GPU memory constraints.
- Rigorous stability and instability conditions are derived.
- The result helps determine if an LLM service can sustain demand without unbounded queue growth.
- Operators can calculate necessary cluster size using arrival rate and stable service rate.
- The paper addresses the core challenge of GPU provisioning.
- KV caching accelerates decoding but exhausts GPU memory.
- The work is from arXiv, published under Announce Type cross.
Entities
Institutions
- arXiv