Edge AI Deployment: Compression Techniques Compared on Real Hardware
A recent study posted on arXiv (2608.15693) examines how to run large AI models on resource-constrained edge devices, focusing on model compression strategies. The authors provide practical advice based on various research findings, testing these strategies on platforms like GPUs, CPUs, and Raspberry Pis for tasks such as answering questions and segmenting images. Interestingly, no single compression technique consistently outperforms the others. For instance, the Qwen3.5 0.8B model achieves a score of 93.85 SQuAD F1 for question answering using Q5_K_M GGUF quantization, whereas structured pruning at the same accuracy leads to a 16 F1 drop at a 1% ratio. In the case of segmentation, pruning can shrink model sizes by nearly 80% without major mIoU impacts, but it may increase the deployed file size by 21-49% due to alignment issues. The paper highlights the need for customized compression methods for different edge AI applications.
Key facts
- Paper arXiv:2608.15693 surveys dozens of recent works on edge AI deployment.
- Experiments conducted on GPU, CPU, and Raspberry Pi platforms.
- Tasks include question answering and image segmentation.
- Qwen3.5 0.8B reaches 93.85 SQuAD F1 and 92 EM under Q5_K_M GGUF quantization.
- Structured pruning at 1% ratio costs 16 F1 for question answering.
- For segmentation, pruning cuts model size by nearly 80% at near-constant mIoU.
- Pruning can inflate deployed artifact by 21-49% due to breaking k-quant super-block alignment.
- No single compression technique wins across tasks.
Entities
Institutions
- arXiv