OmniDrop: Layer-wise Token Pruning for Omni-modal LLMs via Query-Guidance

A novel framework for token pruning, known as OmniDrop, has been introduced for omni-modal large language models. This technique systematically prunes audiovisual tokens within the decoder layers of the LLM instead of at the input stage, tackling the issue of token proliferation resulting from high-resolution audio and video data. Utilizing text queries for guidance, it enables modality-agnostic and task-adaptive pruning, ensuring that earlier layers maintain adequate fusion of omni-modal information before more aggressive token removal occurs in the deeper layers. This strategy seeks to address the shortcomings of current methods that rely on audio-video similarity or temporal co-occurrence. The findings are published in arXiv paper 2605.14458.