BatchLLM Optimizes Large Batched LLM Inference with Prefix Sharing

BatchLLM is a system designed to optimize large batched LLM inference by leveraging global prefix sharing and throughput-oriented token batching. It addresses the limitations of existing LLM inference engines, which are optimized for streaming requests and struggle with large batch tasks that exhibit prefix sharing. Current solutions use LRU-based cache for KV context reuse but suffer from premature eviction and inability to mix decoding tokens with prefill chunks. BatchLLM introduces a global prefix sharing mechanism and a throughput-oriented token batching strategy to improve performance. The system targets offline and large batch tasks common in industry, where throughput is the key performance indicator. The paper is available on arXiv with ID 2412.03594.