GhostServe: Fault-Tolerant LLM Serving via Shadow Checkpointing

GhostServe is a novel checkpointing system designed to protect the key-value (KV) cache in large language model (LLM) inference services from hardware and software faults. As million-token, agent-based applications become more prevalent, the long-running nature of these tasks increases vulnerability to failures, causing costly job interruptions and resource waste. GhostServe addresses this by applying erasure coding to generate parity shards stored in host memory, enabling fast reconstruction of lost KV cache without full recomputation. The system operates in the shadow, meaning it runs transparently alongside the main inference process. This approach ensures seamless resumption of inference after device failures, significantly improving fault tolerance for distributed LLM serving. The work is presented in arXiv preprint 2605.00831.