Samsung Galaxy devices to run multilingual LLMs with dynamic LoRA switching and multi-stream decoding

An innovative framework designed for hardware optimization facilitates effective on-device inference of a multilingual foundation model based on LLaMA for Samsung Galaxy S24 and S25 smartphones, which are powered by SM8650 and SM8750 Qualcomm chipsets, respectively. This system employs application-specific LoRAs as runtime inputs to a singular frozen inference graph, enabling seamless task switching without the need for recompilation or additional memory usage. A multi-stream decoding approach generates various stylistic responses—such as formal, polite, or cheerful—simultaneously in one forward pass, achieving latency reductions of up to six times. To enhance token generation speed, Dynamic Self-Speculative Decoding (DS2D), a tree-based method for predicting future tokens, is utilized. The challenges of deploying large language models on smartphones stem from strict limitations on memory, latency, and runtime adaptability. This research is detailed in arXiv:2604.18655v1.