PTStore: Distributed Prefix Caching for Efficient LLM Inference
PTStore is a distributed prefix caching and replication system for large language model inference, inspired by content delivery networks. It distributes and replicates popular KV cache prefixes across nodes to reduce latency and load imbalance. By decentralizing the KV cache, PTStore enables orders of magnitude larger cache sizes, allowing 5-6x more efficient inference on long passage Q&A datasets compared to baselines that do not aggregate memory across nodes and GPUs.
Key facts
- PTStore is a distributed prefix caching and replication system for LLM inference.
- It is inspired by client caching in Content Delivery Networks (CDNs).
- PTStore distributes and replicates popular tensors forming reusable KV cache prefixes.
- It reduces latency of accessing the KV cache and alleviates load imbalance.
- Decentralization allows expansion of KV cache size by orders of magnitude.
- PTStore executes inferences on long passage Q&A datasets 5-6 times more efficiently than current baselines.
- Baselines do not aggregate memory across different nodes and GPUs.
- Baselines require regenerating the KV cache.
Entities
—