MobileLLM-Flash: Latency-Guided On-Device LLM Design

A novel approach has been unveiled for creating on-device large language models (OD-LLMs) tailored for real-time AI applications on hardware with limited resources. This technique incorporates hardware-in-the-loop architecture search while adhering to mobile latency requirements, allowing for the deployment of models without the need for custom kernels and ensuring compatibility with standard mobile runtimes such as Executorch. Instead of using specialized attention mechanisms, it utilizes attention skipping to enhance long-context processing speed. The method simultaneously refines model architecture (including layers and dimensions) and attention patterns, considering each candidate as a streamlined version of a pretrained backbone with shared weights for effective evaluation. This framework aims for industry-scale implementation, broadening user access through extensive hardware compatibility and near-instantaneous responses.