Latency Prediction Framework for LLM Deployment on Edge Devices
A novel framework has been introduced in arXiv:2607.21602, aimed at forecasting inference latency for large language models on various edge devices. This methodology emphasizes the intricacies of each inference request, incorporating hardware specifics, model variants, and prompts while distinguishing prefill and decode phases. It merges static parameters with real-time hardware data via a gated prediction model. Key considerations include the architecture of the model, prompt characteristics, runtime backend efficiency, and temperature settings. Tests were performed on devices such as Pixel, Jetson Nano, Orange Pi 5 Pro, and RTX 3090, showing marked improvements in latency predictive accuracy for the Pixel 8.
Key facts
- arXiv:2607.21602 introduces a runtime-aware latency prediction framework for LLMs on edge devices.
- The framework represents inference requests as hardware-runtime-model-prompt configurations.
- Inference is separated into prefill and decode phases.
- A gated prediction model adaptively fuses static descriptors with dynamic hardware telemetry.
- Factors affecting latency include model architecture, prompt behavior, runtime backend, hardware utilization, DVFS, and thermal variation.
- Evaluation used Pixel mobile devices, Jetson Nano, Orange Pi 5 Pro, and RTX 3090-class GPU.
- On Pixel 8, total-latency R-squared improved from 0.953 to 0.960.
- Decode-latency R-squared on Pixel 8 improved from 0.957 to 0.973.
Entities
—