SPADE: Speculative Decoding for Efficient Edge-Cloud LLM Inference

SPADE, a novel distributed inference framework, combines speculative decoding between edge devices and the cloud to enhance efficiency and minimize costs and latency while maintaining accuracy. It employs a lightweight draft model on edge devices to swiftly produce candidate tokens, which are then concurrently verified by a larger model in the cloud. Only the accepted tokens are kept, with rejections prompting corrections from the verifier, thereby significantly cutting down cloud queries. This adaptable design transfers most computations to the edge, reducing both inference time and cloud expenses while retaining the larger model's accuracy. This research, identified by arXiv 2608.13076, presents a practical solution to the heavy computational needs of large language models (LLMs), especially in resource-limited edge computing environments.