Decoupled Visual Processing: Efficient Multimodal Adaptation via Modality-Specific Transformer Substitution
Researchers have rolled out an innovative training method known as Decoupled Visual Processing (DVP) aimed at multimodal large language models (MLLMs). This approach replaces the upper layers of a pretrained language model’s decoder with a more efficient transformer block that can be trained separately and is dedicated to handling visual tokens. In the first half of the decoder, both visual and textual tokens are processed together, but then they split: visual tokens move through the new transformer block, while textual tokens continue through the original layers. This design cuts down on computational costs by not requiring all parameters to be fine-tuned, as visual and textual representations diverge in the later layers. More information is provided in a paper on arXiv (2607.26596).
Key facts
- DVP replaces upper decoder layers with a single transformer block for visual tokens.
- Visual and textual tokens are split after the first half of decoder layers.
- The method reduces computational cost for visual instruction tuning.
- The paper is available on arXiv with ID 2607.26596.
- DVP is an efficient training framework for MLLMs.
- The approach leverages the divergence of visual and textual representations in deeper layers.
- The new transformer block is lightweight and independently trainable.
- The framework avoids fine-tuning all parameters of the model.
Entities
Institutions
- arXiv