SpikeMLLM Introduces First Spike-Based Framework for Multimodal Large Language Models to Enhance Energy Efficiency

SpikeMLLM, a novel spike-based framework for Multimodal Large Language Models (MLLMs), addresses computational and energy inefficiencies in existing models by leveraging Spiking Neural Networks (SNNs). The framework overcomes challenges like heterogeneous modalities and high-resolution image inputs through Modality-Specific Temporal Scales (MSTS) and Temporally Compressed LIF (TC-LIF), reducing timestep compression from T=L-1 to T=log2(L)-1. This approach unifies ANN quantization methods in spiking representation space, guided by Modality Evolution Discrepancy (MED), to improve energy efficiency on neuromorphic hardware. Experiments on four representative MLLMs demonstrate its potential for deployment in resource-constrained environments, as detailed in arXiv preprint 2604.18610v1.