LLMVisor: Real-Time Latency Attribution for Multi-Tenant LLM Serving
Researchers have introduced a new model called LLMVisor for analyzing latency in real-time within multi-tenant environments that use large language models (LLMs). As LLMs start running on shared GPU clusters, the practice of co-batching boosts throughput but makes it tricky to monitor individual tenant usage. LLMVisor tackles this challenge using a roofline-guided approach that breaks down memory and compute phases into a clear piecewise-linear format based on FLOPs and memory I/O. It effectively parses batch latency into individual request shares and operates at the microsecond level, making it great for real-time scheduling. Tests were done with Llama 3.1-8B and Qwen 2.5-14B/32B on A100/H100 GPUs, showing significant improvements. You can find the research paper on arXiv under the identifier 2608.08382.
Key facts
- LLMVisor is a real-time latency attribution model for multi-tenant LLM serving.
- It uses a roofline-guided approach to capture memory-bound and compute-bound phases.
- The model decomposes batch latency into additive per-request shares.
- It runs at microsecond scale, suitable for scheduling loops.
- Evaluated on Llama 3.1-8B, Qwen 2.5-14B, and Qwen 2.5-32B models.
- Tested on A100 and H100 GPUs with varying tensor parallelism.
- Compared to token-count baseline, reduces relative error by up to 2.5x at p90 and 3.3x at p99 for prefill.
- Achieves near-perfect R-squared in latency prediction.
Entities
Institutions
- arXiv