Astrolabe: Randomized Prediction-Guided Scheduler for LLM Serving
There’s a new study on arXiv (2508.03611) introducing Astrolabe, a clever scheduler that uses randomized predictions for one-time request management in multi-instance large language model (LLM) services. This scheduler improves load distribution without the issues associated with migration-based rebalancing, which can cause extra overhead and network congestion from KV-cache transfers when demand peaks. Astrolabe utilizes response-length predictions, simulates latency for each instance, and follows a power-of-two-choices method for dispatching requests, effectively preventing request clumping. Tests with the Llama-2-7B/ShareGPT setup show that Astrolabe matches the leading load-aware system's SLO capacity (31.6 vs. 31.5 QPS), while also speeding up the time to first token by 8 to 36 percent and boosting SLO capacity by up to 20 percent during configuration shifts.
Key facts
- Astrolabe is a randomized prediction-guided scheduler for LLM serving.
- It is designed for one-shot request dispatch in multi-instance LLM serving.
- It avoids migration-based rebalancing to reduce KV-cache transfer overhead.
- It uses response-length estimation, simulation-based latency prediction, and power-of-two-choices dispatch.
- On Llama-2-7B/ShareGPT, it matches SLO capacity of best baseline (31.6 vs 31.5 QPS).
- It reduces mean TTFT by 8-36%, P99 TTFT by 16-77%, and mean E2E latency by up to 5.6%.
- It reduces preemptions by about six times once capacity is reached.
- Under configuration shifts, it improves SLO capacity by up to 20%.
- The paper is available on arXiv with ID 2508.03611.
Entities
Institutions
- arXiv